Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use RFC3339 style timestamps in arrow-json #3416

Closed
JayjeetAtGithub opened this issue Dec 30, 2022 · 1 comment · Fixed by #3449
Closed

Use RFC3339 style timestamps in arrow-json #3416

JayjeetAtGithub opened this issue Dec 30, 2022 · 1 comment · Fixed by #3449
Labels
arrow Changes to the arrow crate enhancement Any new improvement worthy of a entry in the changelog

Comments

@JayjeetAtGithub
Copy link
Contributor

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

arrow-json needs to use RFC3339 style timestamps for timestamps with timezones as already done in arrow-csv. We intend to fix the upstream InfluxDB IOX bug https://github.com/influxdata/influxdb_iox/issues/2095.

Describe the solution you'd like

The array_value_to_string function (from arrow-cast) already supports RFC3339 style timestamps. We would use this function inside arrow-json to change to writing RFC3339 style timestamps while writing out record batches to JSON files. The array_value_to_string function will be modified a bit to print a trailing Z, when the timestamp is in UTC.

Describe alternatives you've considered
We could also implement a handle_timestamp helper function as present in arrow-csv inside arrow-json, but that would be a duplication of code.

Additional context
We intend to use array_value_to_string across arrow-csv, arrow-json, and arrow-pretty for consistency in the RFC3339 timestamp implementation in the near future.

@tustvold
Copy link
Contributor

label_issue.py automatically added labels {'arrow'} from #3417

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
arrow Changes to the arrow crate enhancement Any new improvement worthy of a entry in the changelog
Projects
None yet
2 participants