Defense of Lisp Macros

I was a bit baffled the first time I was introduced to Simulink Coder. I get the value proposition: simulate your model so that you have confidence it's doing the right thing, and then essentially just run that model as code instead of implementing it yourself and possibly introducing mistakes. What I didn't understand, as a software guy, not an engineering guy, was why on earth you'd want a graphical programming language to do that. Surely it would be easier to just write your model in a traditional text-based language. Hell, the same company even makes their own language (MATLAB) that would be perfect for the task.

I did a little digging, and it turns out I had it backwards. It's not that Simulink invented a new graphical programming language to do this stuff. Control systems engineers have been studying and documenting systems using Simulink-style block diagrams since at least the 1960s. Simulink just took something that used to be a paper and chalkboard modeling tool and made it executable on a computer.

Without Lisp-like macros, you need preprocessors or code generators. Something like Yacc/Bison is easily done with suitable macros.

In Common Lisp, macros also enable a kind of Aspect Oriented Programming. That's because one can dynamically modify macro expansion using the macroexpand hook. It's a way to modify a code base without changing the source files, which has all sorts of uses.

http://clhs.lisp.se/Body/v_mexp_h.htm#STmacroexpand-hookST

One can implement things natively in Common Lisp, like code coverage tools, that require separate tooling in other languages.

In my experience (Clojure) macro-heavy libraries tend to be powerful but brittle. A lisp programmer can do things in a library that non-lisp programmers can't realistically attempt (Typed Clojure, for example). But the trade offs are very real though hard to put a finger on.

There are several Clojure libraries that are cool and completely reliant on macros to be implemented. Eventually, good Clojure programmers seem to stop using them because the trade-offs are worse error messages or uncomfortable edge-case behaviours. The base language is just so good that it doesn't really matter, but I've had several experiences where the theoretically-great library had to be abandoned because of how it interacts with debugging tools and techniques.

It isn't the macros fault, they just hint that a programmer is about to do something that, when it fails, fails in a way that is designed in to the system and can't be easily worked around. Macros are basically for creating new syntax constructs - great but when the syntax has bugs, the programmer has a problem. And the community tooling probably won't understand it.

> A confirmation of Greenspun's Tenth Rule, if you will.

I've always found this rule suspect.

If it were true you'd have had lispers loudly reimplementing well-known, slowly-and-buggy mid-size C programs in common lisp, at roughly the rateand loudness the Rust people reimplement old C programs in Rust.

Did that actually happen?

If not, this has to be the greatest expression of envy in all of programming language lore.

BTW, for anyone interested in learning more about Lisp macros, Paul Graham's book about advanced Lisp programming, On Lisp, covers the topic pretty extensively and it's freely downloadable from his website:

Book description: https://paulgraham.com/onlisp.html

Download page: https://paulgraham.com/onlisptext.html

So how would we go about switching the entire software industry to use LISP more? I've been struggling with this idea for awhile. It seems that the best languages don't get adopted.

The only consistent explanation I've seen that it is about 'easy'. The other languages have tools to make them easy, easy IDE's, the languages 'solve' one 'thing' and using them for that 'one thing' is easier to than building your own in LISP.

You can do anything with LISP, sure, but there is a learning curve, lot of 'ways of thinking' to adopt the brain to in order to solve problems.

Personally, I do wish we could somehow re-vamp the CS Education system to focus on LISP and other ML languages, and train more for the thinking process, not just how to connect up some Java Libraries.

On that subject, I once made a serious effort at explaining why Lisp macros are so useful:

https://github.com/codr7/whirlisp

Wow this starts strong:

> Replacing Lisp's beautiful parentheses with dozens of special tools and languages, none powerful enough to conquer the whole software landscape, leads to fragmentation and extra effort from everyone, vendors and developers alike. The automotive field is a case in point.

This has been a struggle of mine since day one. Coming from the java world with more formats, tools, IDEs everytime people create a problem was absurd. Lisp was all absorbing.. much reuse on all metalevels.. yet sometimes I believe a bit of "organ"ism is useful.

The book Beautiful Racket is the best defense of macros out there, on "show, don't tell" grounds. Also, it's just so well-written and describes such an interesting set of tools that I'd argue it's worth a skim even if you don't intend to learn the language.

https://beautifulracket.com/

Macros allow you to bring highly tailored DSLs into your code base. You can practice language oriented programming.

It's seductive, but there's a steep cost. You bring in a bunch of new notation, which you have to learn and teach and remember. You have to consider all the interactions that happen when you mix notations. You have to implement all the tooling.

Usually what happens is you don't quite remember what things mean after a while, and you have to study it again. You didn't consider all the interactions, so it isn't that well behaved. And you never got around to implement quality tooling - and there's no obvious place to plug that tooling in, either. And then it turns out that the notation wasn't quite right for implementing the changing requirements of the business.

IMO, someone should kill XCP and A2L. They are in effect, a semihosted debugger. I would rather have semihosted gdb and use .elf. The tooling around A2L is horrendous.

I am not a fan of Lisp. However I sure wish all command line tools would use the same Lisp dialect for scripting. Instead of each Linux tool inventing their own awful, badly documented, makes-you-want-to-vomit bad joke of a scripting language.

Macros are more abstract, one meta level up, hence errors are more difficult to relate to code and reason about. They are also more powerful, so errors can have dramatic consequences.

Any kind of code generation setup will have the same characteristics.

Sloppy macros clashing with local names can lead to pretty long debug sessions, but it's the first thing you learn to avoid.

And you're basically inventing your own ad hoc programming language to some extent, the effort spent on error handling tend to not live up to those requirements.

All of that being said, I wouldn't trade them for anything, and I haven't seen any convincing alternatives to Lisp for the full experience.

I am not reading 10,000+ words on why macros are good. I know macros are good because smart people tell me they are. There is also plentiful evidence of macros being useful to programmers. Who are you trying to convince here?

For such a long and detailed post, not a single lisp expression or macro is visible in this post.

> Replacing Lisp's beautiful parentheses with dozens of special tools and languages, none powerful enough to conquer the whole software landscape, leads to fragmentation and extra effort from everyone, vendors and developers alike. The automotive field is a case in point.

This completely ignores Haskell.

Macros are one of the widely touted superpowers of lisp. And yet in Clojure, which is the most widely used dialect in modern use, it is idiomatic to avoid macros as much as possible. None of the claimed advantages really hold up under scrutiny.

> "I don't think this is truly realized, but all this information written down in word sheets, requirements, knowledge passed on in meetings, little tricks and short hacks solved in video calls, information passed in emails and instant messaging is information that is part of the project, that is put down in some kind of a language, a non-formal one like English or tables or boxes or scribbles or laughs and jokes. The more a project has of these, the harder it is to talk about and verify the actual state of the project as it is obviously impossible to evaluate a non-formal language like you would a formal programming language and thus it becomes that much harder to explore and play with, test and explore the project."

I've thought all this sloppy side information would be a good target for the currently in vogue AI techniques.