-
Notifications
You must be signed in to change notification settings - Fork 237
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[FEA]JSON reader: support "allowBackslashEscapingAnyCharacter" #4616
Comments
Spark supports these escape characters : \", \, \/, \b, \f, \n, \r, \t, \uXXXX. Currently CUDF doesn't throw error for the escape characters which are not supported. Output would be same as input. |
I think falling back to CPU when this option is set is the right way for now. |
Note that this is related to #10596 |
Also I just tested and \uXXXX appears to work properly out of the box with CUDF. |
From the Spark JSON option,
allowBackslashEscapingAnyCharacter
Allows accepting quoting of all character using backslash quoting mechanism.
typically only a few chars in the JSON standard are allowed.
In CUDF they only support ", , \t, \r, and \b. Not sure if there are others in JSON or not. Also not sure what happens if CUDF if others are encountered vs in Spark.
We need to figure out the set of characters which should be supported.
Because of how cuda works it is very rare for CUDF to return an error if it sees an escape it does not understand. We are likely just going to have to document the differences.
The text was updated successfully, but these errors were encountered: