fix(parser): handle nested comments

[yang-han]

I hereby agree to the terms of the Singularity Data, Inc. Contributor License Agreement.

What's changed and what's your intention?

Make the tokenize_multiline_comment() able to handle nested comments, as mentioned in #6518

Checklist

- [ ] I have written necessary rustdoc comments

• I have added necessary unit tests and integration tests
• All checks passed in ./risedev check (or alias, ./risedev c)

Documentation

If your pull request contains user-facing changes, please specify the types of the changes, and create a release note. Otherwise, please feel free to remove this section.

Types of user-facing changes

• SQL commands, functions, and operators

Release note

Users may write nested comments in SQL, e.g., 0/*multi-line\n* \n/* comment \n /*comment*/*/ */ /comment*/1 will be tokenized to

vec![
  Token::Number("0".to_string()),
  Token::Whitespace(Whitespace::MultiLineComment(
      "multi-line\n* \n/* comment \n /*comment*/*/ */ /comment".to_string(),
  )),
  Token::Number("1".to_string()),
];

Refer to a related PR or issue link (optional)

close #6518

[codecov]

Codecov Report

Merging #6526 (b8e3f27) into main (213da6a) will decrease coverage by 0.03%.
The diff coverage is 63.48%.

@@            Coverage Diff             @@
##             main    #6526      +/-   ##
==========================================
- Coverage   74.01%   73.98%   -0.04%     
==========================================
  Files         983      986       +3     
  Lines      160712   161614     +902     
==========================================
+ Hits       118952   119570     +618     
- Misses      41760    42044     +284     

Flag	Coverage Δ
rust	73.98% <63.48%> (-0.04%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
src/batch/src/executor/mod.rs	72.30% <ø> (ø)
src/batch/src/executor/source.rs	0.00% <0.00%> (ø)
src/common/src/util/prost.rs	0.00% <0.00%> (ø)
src/compute/src/rpc/service/stream_service.rs	0.00% <0.00%> (ø)
src/frontend/src/observer/observer_manager.rs	0.00% <0.00%> (ø)
...c/frontend/src/optimizer/plan_node/batch_source.rs	0.00% <0.00%> (ø)
src/frontend/src/optimizer/plan_node/mod.rs	85.49% <ø> (ø)
src/frontend/src/optimizer/plan_node/stream.rs	13.24% <0.00%> (-0.07%)	⬇️
src/frontend/src/scheduler/distributed/stage.rs	19.92% <0.00%> (-1.09%)	⬇️
src/frontend/src/scheduler/local.rs	0.00% <0.00%> (ø)
... and 90 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[lmatz]

Thanks for your contribution! Let's see what the upstream thinks apache/datafusion-sqlparser-rs#726

[lmatz]

LGTM!