-
Notifications
You must be signed in to change notification settings - Fork 915
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix parquet schema interpretation issue #13277
Fix parquet schema interpretation issue #13277
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Thanks for the fix
I've been trying to wrap my head around Parquet's handling for legacy LIST representations as well. I think we are agreed on the first one:
This conforms to the first section of the
So this should be interpreted as Could I please have clarification on the second one?
The backward compatibility rules are:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🚀 Thanks for the fix, @hyperbolic2346. LGTM!
I've re-kicked the failing test. Once it passes, if there aren't any objections, shall we merge this one? |
/merge |
Thank you for this fix, @hyperbolic2346! |
Description
There is a bug reading parquet files that have a specific encoding. The typical list is
but it can also be
This second case was failing and this fixes that. The issue was the
one_level_list
was returning true for the name, which resulted in another nesting level. The fix is to only return true if the immediate parent is not a list.closes #13237
closes #13239
Checklist