Apply NFKC normalization to unicode identifiers in the lexer

[AlexWaygood]

Summary

A second attempt to fix #5003, hopefully without the performance problems that #10381 suffered from.

Python applies NFKC normalization to identifiers that use unicode characters. That means that F821 should not be emitted if ruff encounters the following snippet (but on main, it is), as from Python's perspective, these are all the same identifier:

𝒞 = 500
print(𝒞)
print(C + 𝒞)  # ruff says `C` isn't defined
print(C / 𝒞)
print(C == 𝑪 == 𝒞 == 𝓒 == 𝕮)  # ruff says `C`, `𝑪`, ... isn't defined

This PR fixes that false positive by NFKC-normalizing identifers as they are encountered in the lexer.

Test Plan

cargo test

[github-actions]

ruff-ecosystem results

Linter (stable)

✅ ecosystem check detected no linter changes.

Linter (preview)

✅ ecosystem check detected no linter changes.

Formatter (stable)

✅ ecosystem check detected no format changes.

Formatter (preview)

✅ ecosystem check detected no format changes.

[dhruvmanila]

This is great with such a minimal change. Can you please write / update documentation around this change? Maybe the lex_identifier or the Name token?

It's good to go from my side but I'll let @MichaReiser give the final approval.

[AlexWaygood]

Can you please write / update documentation around this change? Maybe the lex_identifier or the Name token?

Thanks! I had a go at writing some docs in d785be5 -- how does that look to you?

[dhruvmanila]

Thanks! I had a go at writing some docs in d785be5 -- how does that look to you?

Looks good. Thank you!

[charliermarsh]

I was eventually hoping to remove all owned data from the lexer tokens (e.g., prior to this change, we could've conceivably removed this field altogether; if we remove more similar fields from other tokens, we can eventually reduce the size of the Tok enum, which could be great for performance). This now precludes us from doing so. But I don't have enough context on the future design of the lexer-parser to know if it matters.

[MichaReiser]

I think we can still accomplish this and Name isn't the only token that must return the parsed value (e.g. FStrings etc).

What I have in mind is to:

• Replace Tok with TokenKind (that holds no data)
• Add a take_value() method on Lexer that "takes" the current value (enum over Name, Int etc.).

This design also fits better into our new parser that already does exactly this internally (except that it "takes" the value from Tok). The advantage of this is that we only pay the overhead of reading or writting a value if it is a value token (and we're interested in the value).

[dhruvmanila]

This is a good point. We could potentially move this to the parse_identifier method in the parser as that's the final destination for where this value needs to be. I could come back to this once the new parser is merged and I'm looking into the feedback loop between the lexer and parser.

[charliermarsh]

Looks reasonable to me, I'll defer to Micha and Dhruv to approve.

[MichaReiser]

I think we can still accomplish this and Name isn't the only token that must return the parsed value (e.g. FStrings etc).

What I have in mind is to:

• Replace Tok with TokenKind (that holds no data)
• Add a take_value() method on Lexer that "takes" the current value (enum over Name, Int etc.).

This design also fits better into our new parser that already does exactly this internally (except that it "takes" the value from Tok). The advantage of this is that we only pay the overhead of reading or writting a value if it is a value token (and we're interested in the value).

[AlexWaygood]

Thanks all!