Hashtag regex doesn't catch Unicode hashtags

[mojodna]

After 9719a31, Unicode characters (including emoji ;-) in hashtags are no longer included (they're now treated as word separators):

> "#g1b".match(/#[\w-]+/g);
[ '#g1b' ]
> "#g1ü".match(/#[\w-]+/g);
[ '#g1' ]

I've been using /(#[^\d][^#\s,;:]*)/g as my current regex of choice (with the dubious assumption that hashtags shouldn't start with numbers).

[bhousel]

Whoa that is interesting and kind of surprising to me. Lots of info here:
https://stackoverflow.com/questions/280712/javascript-unicode-regexes
https://mathiasbynens.be/notes/es6-unicode-regex

I think, this is fixed in ES6 by adding the /u regex flag. Unfortunately this means it won't work in IE11.

@mojodna, can you submit a PR with a fix that uses one of the ES5-safe options?

[mojodna]

/u doesn't expand the word class, unfortunately. I saw some Stack Overflow posts with suggestions for Unicode ranges that roughly equate to "word", but they seem excessively complex. Inverting punctuation seems promising (potentially with no need for /u support since they're 4 character codes in pairs?). This is from https://stackoverflow.com/a/25575009:

/(#[^\u2000-\u206F\u2E00-\u2E7F\s\\'!"#$%&()*+,\-.\/:;<=>?@\[\]^_`{|}~]*)/g

> "#÷¿øü?,#tag".match(/(#[^\u2000-\u206F\u2E00-\u2E7F\s\\'!"#$%&()*+,\-.\/:;<=>?@\[\]^_`{|}~]*)/g)
[ '#÷¿øü', '#tag' ]

Thoughts?

[bhousel]

@mojodna That seems fine.. I'm ok with a big ugly regex, as long as the code is readable and commented, something like..

// match unicode range and non punctuation, see #4398
var hashtagRegex = /biguglyregex/;   
whatever.match(hashtagRegex);