There's also a PEP to make them first-class: https://peps.python.org/pep-0638/
Syntactic macros are still second-class, like Lisp macros, but an improvement over text-replacement style macros.
For something macro-like which is first-class, there are fexprs[2] and operatives (from Kernel[3]) - these receive their operands verbatim, like macros, so they don't require quotation if we want to suppress evaluation. fexprs/Operatives can be passed around like any other value at runtime.
[1]:https://en.wikipedia.org/wiki/First-class_citizen
Thank you for the clarification and the additional information, I think having macros as first-class objects is a cool (but separate) idea.
I wouldn't say this is the case. Nearly any language could implement an `eval` for itself, but obviously, it is much simpler in Lisps because there's little syntax and few rules.
What makes Lisp macros different from say, C preprocessor macros, is the body of the macro is just Lisp code - so the "preprocessor" in this case has full access to the host languages facilities, which may include `eval`. The macros don't take textual input, but they take structured input in the form of S-expressions.
Implementing macros is obviously simpler due to eval, because we need to run the evaluator on the macro body, but it's not a strict requirement, as macro functionality could be provided by the implementation and could encapsulate its own evaluator.
Lisp macros are also simple due to the fact that Lisp code is just lists of data - you don't have to navigate a complex AST to walk through the code and emit particular syntax. You walk through the input with `car` and `cdr`, and you emit new syntax with `cons` (or `list`/`list*` which are derived from it). Macros can take code as their argument, and produce new code which is evaluated in place.
Macros still have hygiene issues though, because they're based on expanding code before evaluating it, variables used in macros can accidentally shadow variables in the scope of the macro's caller. There are workarounds (gensym) to navigate these hygiene problems.
> From what I gather, Lisp starts with a small set of primitives and special forms—seven in the original Lisp, including lambda. I recall Paul Graham demonstrating in one of his essays that you can build an eval function using just these primitives.
This is largely a theoretical demonstration but not real-world usage. In practice, Lisps have dozens or hundreds of "primitives". Common Lisp in particular is a big language and not trivial to implement. Scheme is a bit smaller, though r6rs started to also grow quite large, but this approach was revisited in r7rs (current), which aims for a small core, with additional functionality being provided through SRFIs (Scheme requests for implementation).
> Those primitives are typically implemented in a host language like C, but once you have an eval function in Lisp, you can extend it with new rules.
Using Scheme as an example, some SRFIs can be implemented purely in Scheme, as libraries, but others require the implementation to provide support, which often requires writing C code to provide them.
> This seems like a way to understand macros, where you effectively add new language rules. I know Lisp macros are typically defined using specific keywords like defmacro
As you note, it's `defmacro`, or `macro`, or `syntax-rules`, `syntax-case`, etc, which introduce new syntax - not eval in particular. Some macros will use `eval` in their bodies, which permits control of evaluation other than the regular applicative form of lambdas.
Macros are more than just `eval`. They're a multi-stage evaluation model where we first need to do some `macroexpand` (which will internally use `eval`), and afterwards the the resulting expression from the macro call is evaluated.
> but is the core idea similar—extending the language by building on the eval function with new rules?
There are some Lisps which still attempt this kind of minimalism.
One example is Ian Piumarta's Maru[1], which support extending `eval` (and `apply`) with new functionality at runtime based on the type being evaluated. Maru basically has global maps of type->evaluator and type->applicator, where we can add new pairs to at runtime and augment the behavior of `eval`/`apply`.
Kernel[2] also aims for the minimalist approach and does away with macros, quote and special-forms entirely, instead replacing them with a more general feature called an operative. The Kernel evaluator does not need to implement special rules for things like `lambda`, `cond`, `car`, `cdr` (as in Graham's "On Lips" essay) - but it just discriminates two forms - operative or applicative. Obviously, some kinds of operative are "primitive", but there's no difference from the PoV of the programmer. Which set of symbols you decide to implement as primitive is up to the implementation. The Kernel report suggests a small set of primitives and demonstrates the remaining standard features can be implemented using only the functionality provided so far.
Seems that what makes strings terrible is that they don't really encode any other structure, already. Contrast this with atoms and lists, which do encode a bit of structure. It isn't just a string of characters, it is a list of atoms and lists. Structure is already part of the thing. It does not have to be parsed out.
If m-expressions would break that, I think I can see your point. But I'm confused on what would be the point of m-expressions, if they are largely just to remove structure from the content.
Calling eval on (1 2 3) gives you exactly what you asked for. It's just probably not what you wanted. The problem isn't with eval, but with the idea that (eval (1 2 3)) or (eval '(1 2 3)) is what you want to do. You probably want (eval ''(1 2 3)) - two single quotes. That'll give you a list containing the first three positive integers.
From the name, though, I would expect that they would still be fully developed expressions. Just as I would not think you would say the string "(1 2 3)" is the same as the list `(1 2 3)`. Specifically, the structure of it being a list of items there is a vital part of the s-expression.
I don't understand the point on calling eval on the list `(1 2 3)`. My point is you can still have a syntax error, even if you pass it a structured list. Nothing more. To that end, it is not valid code. This does assume "is code" meant "is valid code." But, that feels a safe assumption?
For fun, if you don't allow a distinction between the list `(1 2 3)` and the string "(1 2 3)", then you can also flat out eval "(1 2 3)" without error. But that kind of gets to my core point, that the string and the list are different things.
Granted, I fully cede that your point is largely right. Just feels like you are a bit too strong with the "if and only if" on it.
(And yes, that can be used unquoted in a few different contexts in Lisp, such as a special form or macro, or if you've managed to convince the reader that 1 is a function.)
=>(eval '(1 2 3))
Cannot call 1 as a function.
=>(eval ''(1 2 3))
(1 2 3)That said, amusingly, `(1 2 3)` can easily be considered code if you are able to evaluate permutation notation. No?
To be specific, it is data. It can also be code when placed in a context where code is expected (for example in the REPL).
If taken as code and evaluated it will probably raise some kind of exception (saying that '1' isn't a function). Saying that it isn't code because it raises an exception would be an interesting take, considering something like...
(sqrt "Hello, World!")
Data can also be taken literally, as is the case when macros get involved. We do that every time we define a function, for example:
(defun square (x)
(declare (type Number x))
(* x x))
And of course you can have conditional evaluating of data as code, which still makes it code even though it doesn't necessarily get evaluated:
(define (may-error quote-the-thing)
(if quote-the-thing
(quote (1 2 3))
(1 2 3)))
So on the less-well-defined end, you have something like JavaScript's `eval` function, in the middle you have lisp macros, which get an AST instead of just source code, and on the actually-pretty-well-defined end, you have a lambda that's already been parsed/compiled and just needs to be evaluated.
- in Javascript: https://link.springer.com/chapter/10.1007/978-3-642-22655-7_...
(Also, there's a more general problem of newbies trying to use the most advanced tool they've been exposed to. Why use the standard `if` statement, when you can use a proprietary high-powered pattern-matching form, now backed by a networked Kubernetes cluster of deep learning and LLMs.)
In the Racket community, one of my many attempts to discourage mistaken uses of `eval`:
https://groups.google.com/g/racket-users/c/Z-IlF24RAKU/m/3h6...
In the all-in-one practitioner's book I was writing, eval wouldn't be introduced until almost the end, in the "Dangerous Last Resort" part of the book. (Maybe I should've planned a marketing gimmick around it, as "free bonus DLC", and you need to do some ritual to be a Certified Certifiable Racketeer, before you can read the eval secret scroll.)
eval can be sandboxed with a combination of multiple approaches, including validating the code, and evaluating it in a restricted environment in which only certain symbols exist. The full library package is not visible.
dang•8mo ago
On eval in dynamic languages generally and in Racket specifically (2011) - https://news.ycombinator.com/item?id=8098569 - July 2014 (18 comments)