Add Stan Lexer

[jgaeb]

This PR adds a lexer for the Stan Bayesian data analysis language, based on the C and R lexers. Stan files do not have an associated mime type, so I've left that attribute as nil—please let me know if it should be a default text type or something like that. (I figured that would lead to it being chosen as a default parser for text files, but maybe that's not right.)

[jgaeb]

Looks like I didn't take into account functions being called inside other functions. Shouldn't be difficult to resolve (just add another scope for parentheses) but I will be away for a few days and will fix it then.

[spinkney]

Oh I didn't know this already existed!

[spinkney]

@jgaeb can you update for 2.28? This fixes #1768

[jgaeb]

Just added the complex type and complex literals!

[spinkney]

Does this have the array[ ] syntax and will it lex .stanfunctions files?

[jgaeb]

Ah, good catch on the array syntax—that's an easy fix. Can you point me to the spec / something describing the structure of .stanfunctions files? I'm having trouble finding any docs. (Although I'm guessing I'm overlooking something obvious!)

[spinkney]

No worries this is brand new. It should just highlight the same as .stan files but allow the ending to be .stanfunctions.

The doc is at stan-dev/docs#434.

vs-code put it in by just adding .stanfunctions as another extension in the json

[jgaeb]

Cool, both easy changes—should be taken care of now.

[spinkney]

@tancnle I see you've made most of the acceptances. This is ready for review/acceptance

[spinkney]

Also, we need to make sure that complex numbers are highlighted correctly
vscode shows this:

issues:

• complex is not highlighted
• to_complex is not highlighted
• get_real is not highlighted
• and 3i - 40e-3i + 1e10i should each be highlighted as "green", just like regular numbers

[jgaeb]

@spinkney The lexer should deal with all of those issues. Let me know if anything still looks weird to you.

[spinkney]

there's a typo on line 160 for multiply_lower_tri_self_transpose. Will see if there are others

[spinkney]

also, @jgaeb how did you get/make that list of functions/distributions?

[jgaeb]

It looks like the multiply_lower_tri_self_transpose was just a typo. (Edit: Ah, I see you found it too.)

I constructed the list manually from the function reference. I thought a little bit if there might be a way to pull it out of the compiler code or something like that, but I'm not sure where the language's standard functions (as opposed to just the functions defined in the stan/math library) are defined, exactly.

[spinkney]

It looks like the multiply_lower_tri_self_transpose was just a typo. (Edit: Ah, I see you found it too.)

I constructed the list manually from the function reference. I thought a little bit if there might be a way to pull it out of the compiler code or something like that, but I'm not sure where the language's standard functions (as opposed to just the functions defined in the stan/math library) are defined, exactly.

You can get all the signatures this way:

clone the stan-docs repo then cd into it run the python script with the major and minor version numbers:

python3 extract_function_sigs.py 2 28

then run a bash command to extract only the unique names

cut -d ';' -f1 stan-functions-2_28.txt | uniq -u > stan_functions_list-2_28.txt

[spinkney]

@pyrmont @jneen this ready for review

[tancnle]

Thank you for your contribution @jgaeb ❤️ Warm welcome to the Rouge project. stan is an interesting language and it would be great to add this support to Rouge. The PR looks great to me. I have added some minor suggestions for your consideration. Please let me know what you think 🙏🏼

Could you also please add stan to the list of supported languages?

[jgaeb]

@tancnle Thank you very much for the suggestions! I made all of the changes you requested, including to preprocessor directives.

[tancnle]

Thank you for a delightful collaboration on this @jgaeb ❤️ Your work is greatly appreciated 🙏🏼