Convert search terms to NFC

[longrunningprocess]

There may be languages or scenarios where a user will enter a search term and either match/miss entries because of incompatible UNICODE encodings, e.g., NFC or NFD. I'm not 100% sure whether this problem exists in LF or not but thought it would be good to research this possibility more in case there are users out there working in languages and having a lot of difficulty as a result of LF not "normalizing" inputs against the data being searched against.

This thought and a possible situation demonstrating a failing scenario first came up in discussions here: #1243 (comment)

[alex-larkin]

Understanding how different ways of encoding characters and representing words affect language projects

I found through this post a link to the Character Identifier tool that Marc Durdin made. I think we should connect with him to understand this issue better.

[alex-larkin]

Talked with Billy over Meet and here are some thoughts:

• Without failing use cases, this is still somewhat theoretical
• We need to know how FLEx and Language Depot, LF encodes the data. Is NFC or NFD used?

Before going forward, we need

• To talk with @megahirt about how data is encoded in LF (and LD and FLEx)
• Find some failing use cases

One option for search purposes is to normalize both the search query and the data into a cache and then checking for matches. Although constantly normalizing the entire lexicon may tax the resources heavily.

[megahirt]

Some samples of NFD/NFC equivalent strings are in the comment this issue references. There are several examples. When doing a string comparison (byte comparison) you will not get matches for strings that look identical to the user because the underlying codepoints are not the same.

What is curious to me is that our code seems to treat an NFC and NFD string as equivalent so that an exact match search does the right thing. I am digging in further to try and understand.

LF does not normalize data on storage, however my experience tells me that most data as a whole is coming in as NFC. We store whatever the user types using their keyboard. Many keyboards produce NFC, some produce NFD.

@jasonleenaylor does FLEx normalize data when it is stored on disk? I have a feeling it normalizes to NFD, but I'm not certain.

Here is a test character for searching in both NFC and NFD form:

â (NFC)
â (NFD)

These characters are canonically equivalent, look identical, yet contain different code points underneath and will not match when one is typed and the other is present in the data.

(screenshot from the Character Identifier tool)

The actual comparison happens on L478 of entryMeetsFilterCriteria()

Consider the following Regex which demonstrates that an NFD and NFC sequence do not match without special treatment:

// Is "â" (NFC) equal to "â" (NFD) ?
/\u00E2/.test('\u0061\u0302');
// no it is not

I am still investigating why the filter shows a match when the data and search term are not normalized. Update: figured it out. See next comment.

[megahirt]

LF does not normalize data on storage, however my experience tells me that most data as a whole is coming in as NFC. We store whatever the user types using their keyboard. Many keyboards produce NFC, some produce NFD.

I stand corrected. LF does normalize all strings to NFC on import or storage. Here is where it gets converted to NFC. This is why the strings match when I enter in NFD as a data field and then search using an NFC string. This becomes an NFC = NFC comparison.

If I enter an NFD string into the search criteria, the exact match doesn't work as expected, since I now understand what is going on.

This issue should now be defined as: "convert search term to NFC" when searching for exact match. That should do it.

[longrunningprocess]

excellent findings, this is exactly the concrete clarity we needed, thank you @megahirt !

[megahirt]

I have tested this on QA and it is working as expected.