[BUG] \n is not considered whitespace when tokenizing JSON

[revans2]

Describe the bug
CUDF support setting a line delimiter when parsing/tokenizing JSON. But it appears that \r is not considered to be valid white space at least not if json lines parsing is enabled.

Steps/Code to reproduce bug
I need to do some more work to get a repo case in C++. But if I try to parse/tokenize JSON with

{"a":
100}

with the delimiter set to 0, I get back a null indicating that the data is invalid. But if I put the value in quotes

{"a":"
100"}

it is parsed as expected.

Expected behavior
according to the json spec white space include \n, but it does not appear to be treating it that way.

[shrshi]

Thank you for sharing the bug report, @revans2. To clarify, are you setting the delimiter to 0 or \0? If it is 0, then get_token_stream returns error tokens for the three invalid JSON lines.
Also, are you enabling white space normalization for this repro?

[shrshi]

Opened #16923 to fix this bug.

[revans2]

I set the delimiter to the nul character. 0x00. This was meant to be a stop gap because 0x00 is much less likely to show up in a JSON column than \n is.

[ttnghia]

Can this be closed, given that #16923 is merged?

[shrshi]

I believe this can be closed. @revans2 please feel free to re-open if any further issues are encountered.