digester: avoid infinite loop when an invalid token is seen

[kennytm]

What problem does this PR solve?

Starting from pingcap/tidb#10284, executing the following would cause TiDB to enter an infinite loop:

create table `🥳`(a int);
select * from `🥳`;         -- ← this line

This is caused by multiple issues:

1. MySQL does not permit non-BMP characters like 🥳 in an identifier, no matter quoted or not. But TiDB's and MySQL's behavior differ on such non-BMP characters.

   • MySQL allows both unquoted 🥳 and quoted `🥳`, but both will be translated into a question mark `?`.
   • TiDB treats unquoted 🥳 a lexer error, but accepts the quoted `🥳` as-is.

2. The parser.Normalize strips away the backquotes around the identifier, while MySQL adds the backquotes in all cases.

3. Since *: support select/ explain select using bind info tidb#10284, when we execute a SELECT statement, it will call GetBindRecord. The problem is GetBindRecord calls parser.DigestHash on a normalized SQL (*: support "add session binding" tidb#10247), so we will double-normalize the statement before computing the hash.

So, up till now, the statement SELECT * FROM `🥳`; is (1) accepted by the parser, (2) normalized into SELECT * FROM 🥳;, and (3) normalized again before calculating the hash.

4. The unquoted 🥳 is treated as a lexer error. The lexer will just return the invalid token without advancing the cursor. This throws the normalize function into an infinite loop.

What is changed and how it works?

Break the infinite loop if a lexer error is encountered, so at least running SELECT * FROM `🥳`; won't DoS the server.

We could think of how to fix the other 3 problems in other PRs.

Check List

Tests

• Unit test

Code changes

Side effects

Related changes

[kennytm]

PTAL @lysu

[lysu]

LGTM

[jackysp]

LGTM