-
Notifications
You must be signed in to change notification settings - Fork 855
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Infer Parquet JSON Logical and Converted Type as UTF-8 #3376
Conversation
There is a draft PR in apache/arrow#13901 that adds a canonical arrow extension type for JSON data backed by UTF-8 arrays. |
Seems reasonable to me -- I think we ought to have a test for it, however, to avoid regressions in the future |
There is a test? |
@@ -526,6 +526,7 @@ mod tests { | |||
OPTIONAL FLOAT float; | |||
OPTIONAL BINARY string (UTF8); | |||
OPTIONAL BINARY string_2 (STRING); | |||
OPTIONAL BINARY json (JSON); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍 here is the test
Benchmark runs are scheduled for baseline = 9cdc1c1 and contender = a8968cd. a8968cd is a master commit associated with this PR. Results will be available as each benchmark for each run completes. |
Which issue does this PR close?
Relates to #3373
Rationale for this change
JSON data must be UTF-8, we can therefore safely infer json fields as containing UTF8 data. This leads to a superior experience as BinaryArray cannot be output to non-binary formats such as CSV and JSON.
What changes are included in this PR?
Are there any user-facing changes?