Search people's names modulo diacritics, in autocomplete

[maxandersen]

autocomplete seem to not work consistently for characters like àåä

If someone is named ståhle I can't get that to autocomplete it using @sta
but if you are named stéph autocomplete works with @ste

[gnprice]

Thanks @maxandersen ! I agree this would be a useful feature.

It looks like the webapp added this feature earlier this year, in zulip/zulip#11183 / zulip/zulip@b7a0a45f0. Before that, it had similar logic which it used for searching for a user in the right sidebar; we probably also want to use similar logic when searching for a user to e.g. start a PM thread.

A note on how to implement this: this looks like an example of some logic it'd be great to share the code for between web and mobile. For whoever takes this on, see

• c631bdb and preceding commits in src/users/usersActions.js for an example of using code shared from the webapp
• zulip/zulip@a63786a for an example of moving webapp code that wasn't previously shared into the shared package, and zulip/zulip@b70b7df for adding Flow types on it
• docs/howto/yarn-link.md for some mechanics for efficiently developing changes in shared code

In particular:

• remove_diacritics should definitely be shared, and I think it will be easy to share.
• person_matches_query would probably be good to share. It may take some more refactoring work to share, though. So that part should come after fixing this issue.

[rk-for-zulip]

remove_diacritics is greatly simplifiable if we have access to String.prototype.normalize:

export const remove_diacritics = (str: string): string => 
  str.normalize("NFD")
    .replace(/[\u0300-\u036F\u1AB0-\u1AFF\u20D0-\u20FF\uFE20-\uFE2F]/g,'');

(This isn't quite complete; it will notably not remove strokes, such as those found in Ħ, Ł, Đ, or Ð [sic!], nor unify i with Turkish ı. Also, we may or may not want it to unify homographs.)

[timabbott]

zulip/zulip#13485 is an improved implementation we just merged for the webapp for this.

[gnprice]

Neat!

Here's the new version there, which is in a very similar spirit to Ray's comment above:

const unicode_marks = /\p{M}/gu;

exports.remove_diacritics = function (s) {
    return s.normalize("NFKD").replace(unicode_marks, "");
};

At that length it'd feel less bad to me to duplicate it. ... But also that change itself illustrates why it'd be better to share it. (The possibility we might add more normalization in the future -- so e.g. "Luk" finds "Łukasz", following on an example above -- is another reason.) Since it should be an easy one to share, I think that'll still be the preferred way to solve this issue.

[showell]

remove_diacritics is now in static/shared/js/typeahead.js.