Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

json top level list scoping #17575

Draft
wants to merge 8 commits into
base: branch-25.02
Choose a base branch
from
Draft

json top level list scoping #17575

wants to merge 8 commits into from
+160 −15

Conversation

karthikeyann
Copy link
Contributor

@karthikeyann karthikeyann commented Dec 11, 2024 •
edited
Loading

debug prints, for list, struct, str mixed together case - updated condition
unit test from issue,
a few tweaks to get unified_schema correctly for mark_is_pruned condition for mark_is_pruned pass (attempted, not fixed)

Related to NVIDIA/spark-rapids#11717

Description

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@karthikeyann
skip array of arrays if experimental
85dda05 
debug prints, commented a few error lines (for list, struct, str mixed
together case)
unit test from issue, (INT32 is still returned as string)
a few tweaks to get unified_schema correctly for mark_is_pruned
condition for mark_is_pruned pass (attempted, not fixed)
@karthikeyann karthikeyann added 0 - Waiting on Author Waiting for author to respond to review 5 - DO NOT MERGE Hold off on merging; see PR for details breaking Breaking change labels Dec 11, 2024
@copy-pr-bot copy-pr-bot
Copy link

copy-pr-bot bot commented Dec 11, 2024

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@github-actions github-actions bot added the libcudf Affects libcudf (C++/CUDA) code. label Dec 11, 2024
@karthikeyann karthikeyann requested a review from shrshi December 11, 2024 04:31
@karthikeyann karthikeyann mentioned this pull request Dec 11, 2024
Open
@karthikeyann
Copy link
Contributor Author

karthikeyann commented Dec 11, 2024 •
edited
Loading

@shrshi I fixed the code and the unit test passes.
It's possible to mix [] and {} types. If the requested schema is list/struct, it will keep only respective type under the top parent.

@revans2 is testing this PR with spark requirements.

@karthikeyann
Copy link
Contributor Author

/ok to test

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@shrshi shrshi Awaiting requested review from shrshi

At least 1 approving review is required to merge this pull request.

Assignees

@karthikeyann karthikeyann

@shrshi shrshi

Labels
0 - Waiting on Author Waiting for author to respond to review 5 - DO NOT MERGE Hold off on merging; see PR for details breaking Breaking change libcudf Affects libcudf (C++/CUDA) code.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@karthikeyann @shrshi