Hygienic macro proposal

[vanviegen]

Here's the macro system we've been using internally. It's not very lispy, but that might be a good thing to most. :) I'd say it's pretty clean, well, for a macro system anyway...

Some examples. Optionally compiled debug info:

macro debug (args...) -> new macro.Call(new macro.Literal("debugImpl"), args) if @debugging
macro -> @debugging = true
debug "test", 123 # console.log gets generated
macro -> @debugging = false
debug "test", somethingHeavy() # nothing gets into the output js

The mandatory swap example. Cheating a bit by using deconstruction, but this does show some of the more interesting features. The coffeescript code "[x,y] = [y,x]" get translated to nodes, and then an identifier substitution replaces x and y by whatever nodes are the arguments.

macro swap (a,b) -> (macro.codeToNode -> [x,y] = [y,x]).subst {x:a,y:b}
swap var1, var2
swap d[123], e[f].g

If you want, you can go pretty crazy with this. Non-sense from the unit test:

tst = (a,b) -> a*b
tst2 = -> 4
macro calc (c1,c2,c3,c4) ->
  func = macro.codeToNode ->
    x = (a) -> tst(a,c1) * c2
    x(c3) + x(c4)
  func.subst {c1,c2,c3,c4}
eq 144, calc 6, 3, 5, 3
eq 144, calc (macro.codeToNode -> tst2()+2), 3, 5, 3

One thing we're using compile-time logic for, is resource versioning of web resources (so we can have clients cache them permanently, and all references to them will automatically change the (otherwise ignored) version number whenever the resource is updated.

macro -> @getFileVer = (file) -> 13 # stub
macro resUrl (name) ->
    name = macro.nodeToVal name
    macro.valToNode "/data/#{name}?v=" + @getFileVer("static/#{name}")
someImg.src = resUrl 'test.png'

Also one of the more interesting applications (for us), is string translations.

interpolateAndPluralize = (msg,arg) -> msg.replace("%1",arg).replace(/[\[\]]/g,'') # stub

macro -> @transDict = waterBottles: "%1 bottle[s] of water" # can be loaded from a file

macro trans (args...) ->
  args[0] = macro.valToNode @transDict[macro.nodeToId args[0]]
  new macro.Call(new macro.Literal("interpolateAndPluralize"), args)

someExample = trans waterBottles, 17

In this example, we're just doing the actual translation compile-time, but one could easily improve upon this by resolving the string insertions at compile time, and generating custom code for the pluralizations.
One could even accept the coffeescript string interpolation syntax. Eg:

console.log trans "I'm to be translated within #{days} day[[s]]!"

would translate to:

console.log "Vertaal mij binnen #{days} dag#{days==1?'':'en'}!"

I'm quite curious to hear any thoughts on this approach...

Regards,
Frank

[vendethiel]

Impressive piece of work.

[vanviegen]

@Nami-Doc, thanks!

Two things I forgot to mention:

• This patch is loosely based on Luc Fueston's work: https://github.com/mrluc/macros.coffee
• If you want to play with it, don't forget to run with the 'coffee --macro' flag, or it won't work as well.. :)

[jashkenas]

Really, really interesting. Something to mull over.

[robotlolita]

Can we drop the csToNode function? I'm pretty sure people will misuse it as @csToNode("#{stuff} = #{other}") and all hell will break loose :(

But this is an interesting piece indeed. I wonder if there's a way to provide a simpler way for people to generate the structure they want inside a macro without using Strings, which are fairly error-prone and difficult to debug. Lisp is at a clear advantage here because the language is basically a huge AST ;)

[STRd6]

This seems pretty interesting, but I'm not sure how it would work for the specific rare cases where I've wanted macros.

How would I use this to implement an infix a + b operator that would translate into (a).add((b))?

[vanviegen]

@killdream: Indeed! Please take a look at the commit I just pushed. I updated the examples in my initial pull request too. Better?

Also, using @ as the namespace for macro helpers felt like a kludge. I'm using the macro namespace on root now. Additionally, macros can now have dots in their names (mostly to accomodate for codeToNode). Oh, and the macro keyword is now a macro itself... I'm sure their must be ways to have some fun with that! ;)

@STRd6: Operator overloading in a dynamically typed language sounds pretty scary. You would end up having to emit extra code for every addition you are doing, in order to check if the operands happen to be of the types you're interested in. This pull request only concerns macros with a function-call syntax.

[STRd6]

@vanviegen I wasn't speaking of operator overloading, but of defining the + operator for a file or section of code. This would be handy for doing math with points or complex numbers. I'm not sure what the point of function-call style macros are since one already has full access to higher-order functions at runtime.

[vanviegen]

@STRd6: Ah, I see that you're into game programming, so you're after vector/matrix arithmetic - I can relate to that. :) It turns out this is actually pretty easy to do: (although this solution does rely on coffee's AST not changing too much)

macro delegateArithmetic (func) ->
    operators =
        '+': 'add'
        '-': 'sub'
        '*': 'mult'
    macro.walk func.body, (node) ->
        if node instanceof macro.Op and (op = operators[node.operator])
            macro.csToNode("(a).#{op}(b)").subst {a: node.first, b: node.second}

# this uses regular javascript operators:
eq 7, 3+4
eq "3,45,6", [3,4]+[5,6]

delegateArithmetic ->
    # this uses the arithmetic methods defined on the prototypes:
    eq [3,4,5], [1,2,3]+2  
    eq 20, [1,2,3]*[2,3,4]
    eq [1,0,1], [3,2,4]-[2,0,3]
    eq 9, 3*3

The generated javascript looks like this:

eq(7, 3 + 4);
eq("3,45,6", [3, 4] + [5, 6]);

eq([3, 4, 5], [1, 2, 3].add(2));
eq(20, [1, 2, 3].mult([2, 3, 4]));
eq([1, 0, 1], [3, 2, 4].sub([2, 0, 3]));
eq(9, 3..mult(3));

[connec]

Haxe has an interesting macro system. Rather than using a csToNode/'evalish' equivalent you can build up an AST using the nodes directly, e.g. your above example might look something like:

...
    macro.walk...
        ...
            new macro.Call new macro.LiteralAccess(node.first, new macro.Literal(op)), node.last

They also have a macro keyword than can be used in macros to convert an expression to an AST, and a makeExpr macro to convert an actual variable to an AST expression for the variable e.g.:

macro do_thing ->
  ast   = macro do ->
  thing = -> console.log 'hello'
  ast instanceof macro.Do # true
  ast.target = macro.makeExpt thing

do_thing() # logs 'hello'

Obviously this conflicts with the macro context, but it's a neat idea.

[vanviegen]

@connec: the new macro.Call etc way of doing things already works, almost exactly like you describe. I think it would be best to use that as little as possible though, as it depends on the structure of the AST, which I'd say should be considered an implementation detail of the Coffee compiler.

The keyword to convert an expression to an AST is called macro.codeToNode. It takes a closure without parameters, and returns the AST of its body.

I'm not really sure if it would be possible to implement a makeExpr macro or function like you describe, as it seems to require some magic to convert a javascript function to a coffeescript AST. There is however a macro.valToNode function that does just this, but only for jsonable expressions.

[ngn]

@vanviegen Thanks for this pull request, it's a dream come true.

Is it possible for one .coffee file to use macros from another .coffee file?

Is the following behaviour of subst intentional?

macro len (x) -> (macro.codeToNode -> x.length).subst {x}
len a   # generates just "a"

macro len (x) -> (macro.codeToNode -> (x).length).subst {x}
len a   # generates "a.length"

[vanviegen]

@ngn, that was a bug, thanks for reporting it! Fixed, and added a few cases to the unit test.

[ngn]

Great!

Now let me elaborate on the other problem I mentioned above: From a.coffee I want to be able to use macros defined in b.coffee, regardless of the order in which those files are compiled, and regardless of whether they are compiled together or separately. Currently macros are usable only from the file where they are defined.

One solution might be to have a built-in macro that imports all macro definitions from another file:

# a.coffee:
requireMacrosFrom 'b.coffee'
myMacro 1, 2, 3

# b.coffee
macro myMacro (x, y, z) -> ...

It wouldn't be too hard to implement outside the compiler:

macro requireMacrosFrom (f) ->
  ast = macro.fileToNode macro.nodeToVal f
  # ... walk the ast and keep only macro definitions
  return ast

but it's only useful as a built in since we can't afford repeating this code in the beginning of every other coffee file.

Any other ideas?

[vanviegen]

@ngn, I think it would be cleaner to just have a separate file that defines common macros. You could include it from both a.coffee and b.coffee.

# common-macros.coffee
macro TEST -> macro.valToNode 42

# a.coffee
macro -> macro.fileToNode 'common-macros.coffee'
eq TEST(), 42

It might be even better if you would be able to specify a common prefix file on the coffee compiler cli. I don't think it can do that at present. We use our own little cli that accepts multiple input files to be combine into a single js file. Check out my blackcoffee2 branch if you're interested.

[ngn]

Ah, I didn't realize that file inclusion can be made as brief as macro -> macro.fileToNode 'common-macros.coffee'. This approach sounds good enough for me to start using macros in practice, then. Thanks.

[ngn]

I noticed that macros in switch statements don't get expanded.

[vanviegen]

@ngn, fixed, thanks!

[jashkenas]

@vanviegen

Are you willing to maintain this as a fork with a nice README for a bit? If so, I'd love to start linking folks over to you from the homepage so you can get more feedback...

[vanviegen]

@jashkenas, allright, how's this? https://github.com/paiq/blackcoffee

[xkxx]

I played with blackcoffee a little bit, and opened a new pull request on paiq#1 for adding method macros in the form of element.$toggleClass "icon-smile". Would someone please review it, thanks.

[mrluc]

I just found my way here via linkage in comments in Hacker News. I had not heard about this; I live in (remote-ish) South America 9 months out of each year so ... I miss things.

Wow! +100 @vanviegen. Thanks for the nod, too -- macros.coffee was a labor of love that started when I wondered if I could get 'quote' to work with just AST fiddling (are you guys still using deepcopy on the AST?), but I've been waiting (and hoping) for a 'heavy-duty' implementation of macros from somewhere, especially one that has decided on an approach to compilation (which I broke in a rewrite and never revisited). I'd love nothing better than for quality macros to happen in Coffeescript. It's already the most beautiful, fluid and convenient language out there.

@jashkenas - I love pretty much all of your code, but mostly I love Coffeescript. I want to use it for everything - and I really think that macros are an approach that would let people use it for things no one can imagine right now.

Also, since there are apparently macro-loving Coffeescripters in this thread, can I just say: sorry. I've had zero time to work on macros.coffee in the past year and more. Like I say on the project page, for anyone who makes more of a 'real' implementation, I'd love to just use yours.

Some implementation thoughts that may or may not still be helpful:

Besides not having time, a somewhat paralyzing consideration was the fact that, at the end of the day, my approach relied on copying an AST and direct fiddling with its nodes in a way that depends on compiler implementation details, and that unedifying work to 'insulate' ourselves from the fragility of that approach would probably be required to move forward in a serious way -- either by creating a compiler from AST <-> our own representation, or by hiding the fragility behind a library of common AST functions (which, when updated for new versions of coffeescript, could insulate macro authors from changes in the CoffeeScript AST api).

Important notes:

• The 'ast' for Lisp code is not a living, breathing organism in the way that ASTs in other languages are (ie, in CoffeeScript and most other languages, each kind of node has behaviors and, basically, its own 'api', whereas Lisp code is represented by lists of basic value types). I see this is alluded to on the BlackCoffee fork's page, and yeah -- it's a tough one.
• It's a testament to the cleanliness of CoffeeScript's code that we can just deep-copy bits of AST and have it all still work. Are you guys still doing that? @jashkenas I really think that if Coffeescript could support macros in a minimal way just by continuing to have an AST that can be deep-copied in whole or in part and still work.

I haven't checked the implementation much at all, I just saw the notes on HN a few minutes ago and had to say thanks, and gush, and wish you all the best, so my comments above might not be relevant to your implementation.

Macros + Coffeescript development. Extremely exciting. I'll help if I can.

[Charuru]

Can't believe JS lived this long without macros. Excited to see where blackcoffee goes.

[rlidwka]

Can't believe JS lived this long without macros.

http://sweetjs.org/ ?

[mrluc]

@Charuru +1 the excitement, though this PR is about CS, not JS. There have been several projects that add macros to javascript before (even before sweetjs, which adds hygenic macros).

Those projects, which are very tied to fiddling with Javascript syntax, and which in HN discussion were talked about by some people as a way of building in 'some of the bits of coffeescript that I like into javascript, but not the rest', would not be as interesting to people who like Coffeescript.

Many people like Coffeescript, find it to be an absolutely joyous and natural language to use, and are interested in macros, perhaps partly for aesthetic reasons, but mostly because macros give you new capabilities, and they want to have those capabilities available to them.

[ai10]

+1

[GotEmB]

[mizchi]

👍
I think coffeescript is one of wise preprocessor, not a mere programming language.

I feel coffeescript project is a bit conservative to ES6. I often want escape hatch about them on syntax level like sweet.js.

Macro may solve them.

[ThaisRobba]

This is seriously incredible. 👍
The capabilities for file inclusion alone are awesome.

[GeoffreyBooth]

As awesome as this is, it doesn’t look like this is headed into the compiler anytime soon. I think an approach that might pass muster is if the compiler is updated to allow plugins somehow, and then this could be released as its own project as a plugin. (The parsing of Markdown in Literate CoffeeScript would make another excellent candidate to extract into a plugin.)

I’ve added BlackCoffee and a link to this PR to the List of languages wiki page.

[zolomatok]

Could we please please please have blackcoffee merged into coffeescript? 🥺

I specifically heard Jesus mention he's not coming back until we have macros (and infix operators on custom objects) in coffeescript. For realsies.

I'll forgo all other presents if I can just have macros for christmas. 🎁

[GeoffreyBooth]

Could we please please please have blackcoffee merged into coffeescript?

It hasn't been updated in 8 years, which means it's a fork of CoffeeScript 1 rather than 2. So unless blackcoffee catches up to trunk, it can't be merged in (on a technical level, aside from whether folks agree that it's a good idea).

[Charuru]

Could we please please please have blackcoffee merged into coffeescript?

It hasn't been updated in 8 years, which means it's a fork of CoffeeScript 1 rather than 2. So unless blackcoffee catches up to trunk, it can't be merged in (on a technical level, aside from whether folks agree that it's a good idea).

I don't think he means literally more that the macros are implemented into coffee. I really want this too, I've since started using TS but if coffee gets macros...

[zolomatok]

It hasn't been updated in 8 years, which means it's a fork of CoffeeScript 1 rather than 2. So unless blackcoffee catches up to trunk, it can't be merged in

Thanks so much for the reply!
Oh, yeah, you’re right! I wonder if it’s a lot of work to bring it up to snuff? To be honest, I’m mostly of course just longing for macros, I’m not sure if there’s a simpler way than modernising blackcoffe.

aside from whether folks agree that it's a good idea

I really, really hope they do. From the looks of it, the reception to this issue was very warm.

[zolomatok]

I don't think he means literally more that the macros are implemented into coffee.

Yeah, I’m just wishing for those tasty macros. Especially for operators.

I really want this too, I've since started using TS but if coffee gets macros...

Me too :D I miss CS so much though. I thought the type system would be enough of a carrot, but alas, no. I miss the expressivity and cleanliness of CS.

[STRd6]

@zolomatok and @Charuru if you like CoffeeScript and TypeScript you may want to check out https://github.com/DanielXMoore/Civet. It's currently in active development and aims to merge the benefits of both CoffeeScript and TypeScript. It doesn't yet have hygenic macros but I'm not fundamentally opposed to the idea and I would be happy to explore it with your feedback.

[zolomatok]

@STRd6 Very intriguing, thanks! 👏
(sorry for the late response I couldn't respond before)

It's awesome that this is a thing! I was happy to see the VSC extension as well!
I had some trouble with the compiler, I opened an issue.

I would love to see macros and infix operators on custom objects in Civet 🌈
Should I open an issue on the repo for that?