Use regex in addition to re?

[julienmalard]

Suggestion
While using Lark I came across an already-documented bug in the builtin re module, whereby abugidas with vowel marks (that is, most of South Asia and Oceania) fail to match the \w directive (e.g., as in re.match("^[\w\s][\w\s]*", "किशोरी", re.UNICODE), as taken from this question here).

It seems that the recommended "solution" is to simply used the regex module instead of re. However, this would mean adding a dependency to Lark, so I thought I would ask whether you would look favourably upon such a suggestion before submitting a pull request. I was thinking of proposing something along the lines of:

try:
    import regex as re
except ImportError:
    import re

so that regex is not required and nothing breaks if the dependency is not installed.

Describe alternatives you've considered

An alternative, which I am currently using as a workaround, is to generate the toxens manually. For instance, for a Python parser, I can use:

ID_START = ''.join([
    c for c in map(chr, range(sys.maxunicode))
    if category(c) in ['Lu', 'Ll', 'Lt', 'Lm', 'Lo', 'Nl']]) + '_'

ID_CONTINUE = ID_START + ''.join([
    c for c in map(chr, range(sys.maxunicode))
    if category(c) in ['Mn', 'Mc', 'Nd']]) + '·'

and then define my NAME token as ID_START ID_CONTINUE*. However, I think that it would be much nicer to be able to use \w according to Unicode standard. A side benefit of using regex is that projects using Lark and specifying regex as a dependency could also then explicitly use Unicode categories in their regexes as well.

Please do let me know!

[erezsh]

Hi,

I'm certainly open to supporting the regex library, if it's useful.

Is it fully-compatible with re, or are there differences beyond bugfixes?

How does it compare in terms of performance?

I think it might be nice to support something like

pip install lark-parser[regex]

Not sure about making it the default though.

[julienmalard]

Hello,
Thank you! I am not sure about performance; if you have a few recommended benchmarks for common regexes in grammars I could check quite quickly though.
I am quite sure that it is fully compatible. Apart from bugfixes, it also allows for unicode categories (see comments here).

If you like, I could make a pull request and we could then check performance, etc. from there?
Thanks!

[erezsh]

Sure, that sounds like a good idea!

[julienmalard]

Thanks! Done in #593.

[julienmalard]

Closed by #593.