Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] ScanJson and JsonToStructs don't process escapes in key names #10607

Open
revans2 opened this issue Mar 15, 2024 · 1 comment
Open

[BUG] ScanJson and JsonToStructs don't process escapes in key names #10607

revans2 opened this issue Mar 15, 2024 · 1 comment
Labels
bug Something isn't working

Comments

@revans2
Copy link
Collaborator

revans2 commented Mar 15, 2024

Describe the bug
It looks like escape processing is not being done on key names, which results in us potentially missing data. This is a little hard to reproduce because of how scala does escape processing, but...

Seq("{\"\\u0064\\u0061t\\u0061\": \"TEST\"}").toDF("value").repartition(1).selectExpr("*", "from_json(value, 'data string').data").show(false)

produces the result 'TEST', on the CPU, but null on the GPU.

@revans2 revans2 added bug Something isn't working ? - Needs Triage Need team to review and classify labels Mar 15, 2024
@revans2 revans2 mentioned this issue Mar 15, 2024
62 tasks
@revans2
Copy link
Collaborator Author

revans2 commented Mar 15, 2024

I am not sure what is happening, it works for me some of the time, but not all of the time, and I don't know why. I will keep digging.

@mattahrens mattahrens removed the ? - Needs Triage Need team to review and classify label Mar 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants