Adds support for json lines format to the nested JSON reader #11534

elstehle · 2022-08-15T14:35:42Z

This PR adds support for the json lines (aka newline-delimited json) format to the nested JSON reader.

Checklist

I am familiar with the Contributing Guidelines.
New or existing tests cover these changes.
The documentation is up to date with these changes.

Thanks to @upsj, who made the translation easier to read for all of us 🙏

…fea-read_json-experimental

Co-authored-by: Bradley Dice <[email protected]>

…cudf into fea-read_json-experimental

…fea-read_json-experimental

…exp-read_json-adapter

…ted-json-lines

codecov · 2022-08-15T16:32:04Z

Codecov Report

❗ No coverage uploaded for pull request base (branch-22.10@c666d7c). Click here to learn what that means.
The diff coverage is n/a.

❗ Current head 16c62cd differs from pull request most recent head 9822ecb. Consider uploading reports for the commit 9822ecb to get more accurate results

@@               Coverage Diff               @@
##             branch-22.10   #11534   +/-   ##
===============================================
  Coverage                ?   86.41%           
===============================================
  Files                   ?      145           
  Lines                   ?    22959           
  Branches                ?        0           
===============================================
  Hits                    ?    19839           
  Misses                  ?     3120           
  Partials                ?        0

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

…exp-read_json-adapter

elstehle · 2022-08-24T08:24:07Z

Looks good, but tests could be beefed up, given the size of the feature. Did not review the state machine :)

Agree 👍 I've added a couple more tests to this PR now. Planning to add more extensive testing once we also cover type inference and type casting

galipremsagar

We can merge the two tests into one with parametrization and use BytesIO for quicker IO than touching the disk.

python/cudf/cudf/tests/test_json.py

…ted-json-lines

karthikeyann

Added some comments for documentation.
Rest all looks great 👍 🚀

cpp/src/io/json/nested_json.hpp

karthikeyann · 2022-08-25T11:07:22Z

cpp/src/io/json/nested_json_gpu.cu

@@ -468,256 +471,411 @@ auto get_transition_table()
 auto get_translation_table()
 {
  std::array<std::array<std::vector<char>, NUM_PDA_SGIDS>, PD_NUM_STATES> pda_tlt;


on a lighter note, to reduce verbosity, this could be added only inside this function
to get rid of repeated typing of token_t::

constexpr auto StructBegin = token_t::StructBegin; constexpr auto StructEnd = token_t::StructEnd; constexpr auto ListBegin = token_t::ListBegin; constexpr auto ListEnd = token_t::ListEnd; constexpr auto StructMemberBegin = token_t::StructMemberBegin; constexpr auto StructMemberEnd = token_t::StructMemberEnd; constexpr auto FieldNameBegin = token_t::FieldNameBegin; constexpr auto FieldNameEnd = token_t::FieldNameEnd; constexpr auto StringBegin = token_t::StringBegin; constexpr auto StringEnd = token_t::StringEnd; constexpr auto ValueBegin = token_t::ValueBegin; constexpr auto ValueEnd = token_t::ValueEnd; constexpr auto ErrorBegin = token_t::ErrorBegin;

and follow similar column like arrangement above.
only issue is that, some symbol groups have multiple outputs, so, indentation will not be regular.
so, you could wrap around clang-format offand clang-format on to keep the code's custom alignment (for eg. 5 entries per row)

@bdice for suggestions

Thanks, I've shortened the names.

Is there a way to allow people to automatically reformat after they modified the translation table? Because we do want to keep in mind that this is a living piece of code. We pursued this approach because it is flexible and can be adapted to meet current and future feature requests. That means the translation table will be subject to change and additions. I would like to keep barrier to contributions low and avoid people having to fiddle with manually inserting tabs and whitespaces. That's why we commented the table to the best of our abilities.

cpp/tests/io/nested_json_test.cpp

cpp/tests/io/json_test.cpp

cpp/src/io/json/nested_json_gpu.cu

karthikeyann

LGTM 👍 🚀

elstehle · 2022-08-25T20:34:30Z

@gpucibot merge

Fixes compile warning introduced in #11534 ``` /cudf/cpp/src/io/json/nested_json_gpu.cu(970): warning #177-D: variable "single_item_count" was declared but never referenced ``` Removed unreferenced variable declaration. Authors: - David Wendt (https://github.com/davidwendt) Approvers: - Nghia Truong (https://github.com/ttnghia) - Bradley Dice (https://github.com/bdice) URL: #11607

Fixes compile error introduced in PR #11466 due to mismatched changes occurring in PR #11534 https://gpuci.gpuopenanalytics.com/job/rapidsai/job/gpuci/job/cudf/job/prb/job/cudf-cpu-cuda-build/CUDA=11.5/11851/console Authors: - David Wendt (https://github.com/davidwendt) Approvers: - Elias Stehle (https://github.com/elstehle) - Karthikeyan (https://github.com/karthikeyann) - Nghia Truong (https://github.com/ttnghia) URL: #11637

vuule and others added 13 commits July 22, 2022 11:51

add placeholder experimental JSON reader

9f2247f

doc fix

76b2834

copyright year

f5464f6

Merge branch 'branch-22.08' of https://github.com/rapidsai/cudf into …

fa2c3a6

…fea-read_json-experimental

newline

2ca0ac0

Co-authored-by: Bradley Dice <[email protected]>

Merge branch 'fea-read_json-experimental' of http://github.com/vuule/…

7ee4b41

…cudf into fea-read_json-experimental

use span

3ee7a5a

Merge branch 'branch-22.08' of https://github.com/rapidsai/cudf into …

fa9ab89

…fea-read_json-experimental

options check + decompression

fcc90c5

Merge branch 'branch-22.10' of https://github.com/rapidsai/cudf into …

b481d66

…exp-read_json-adapter

Merge branch 'branch-22.10' of https://github.com/rapidsai/cudf into …

bdbc111

…exp-read_json-adapter

adds support for ndjson

22b5a46

Merge remote-tracking branch 'upstream/branch-22.10' into feature/nes…

907a846

…ted-json-lines

github-actions bot added the libcudf Affects libcudf (C++/CUDA) code. label Aug 15, 2022

elstehle added feature request New feature or request 2 - In Progress Currently a work in progress cuIO cuIO issue non-breaking Non-breaking change labels Aug 15, 2022

elstehle added this to the Nested JSON reader milestone Aug 15, 2022

elstehle and others added 2 commits August 15, 2022 10:48

addresses outstanding todo

87fce7d

Merge branch 'branch-22.10' of https://github.com/rapidsai/cudf into …

c41f178

…exp-read_json-adapter

elstehle marked this pull request as ready for review August 15, 2022 19:22

elstehle requested a review from a team as a code owner August 15, 2022 19:22

elstehle requested review from vyasr and nvdbaranec August 15, 2022 19:22

vuule added 3 commits August 15, 2022 16:39

C++ side changes + test

9669c6a

working Python + test

c9fb5b2

clean up

2de91e1

elstehle added 2 commits August 23, 2022 21:58

fixes code comments

c6f8d0e

adds more tests for json lines

e38f3d8

adds json lines test for experimental nested json reader

cdb743d

github-actions bot added the Python Affects Python cuDF API. label Aug 24, 2022

fixes style

713260f

elstehle requested a review from galipremsagar August 24, 2022 19:03

galipremsagar approved these changes Aug 24, 2022

View reviewed changes

python/cudf/cudf/tests/test_json.py Outdated Show resolved Hide resolved

galipremsagar self-requested a review August 24, 2022 19:18

elstehle added 2 commits August 24, 2022 12:59

parametrizes test and uses bytesio

14749f7

adds seek before reads

94daa4f

galipremsagar approved these changes Aug 24, 2022

View reviewed changes

ttnghia approved these changes Aug 25, 2022

View reviewed changes

Merge remote-tracking branch 'upstream/branch-22.10' into feature/nes…

310f519

…ted-json-lines

karthikeyann reviewed Aug 25, 2022

View reviewed changes

elstehle added 4 commits August 25, 2022 06:26

prettifies translation table

c09c4af

default_stream and more constness

6efecf4

add TODO for stack ctx interface

272bc16

clarifies treatment of empty lines for ndjson

9822ecb

elstehle requested a review from karthikeyann August 25, 2022 16:06

karthikeyann approved these changes Aug 25, 2022

View reviewed changes

rapids-bot bot merged commit c8c8025 into rapidsai:branch-22.10 Aug 25, 2022

davidwendt mentioned this pull request Aug 25, 2022

Fix compile warning in nested_json_gpu.cu #11607

Merged

3 tasks

davidwendt mentioned this pull request Sep 1, 2022

Fix compile error in benchmark nested_json.cpp #11637

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds support for json lines format to the nested JSON reader #11534

Adds support for json lines format to the nested JSON reader #11534

elstehle commented Aug 15, 2022 •

edited

Loading

codecov bot commented Aug 15, 2022 •

edited

Loading

elstehle commented Aug 24, 2022

galipremsagar left a comment

karthikeyann left a comment

karthikeyann Aug 25, 2022

karthikeyann Aug 25, 2022

elstehle Aug 25, 2022 •

edited

Loading

karthikeyann left a comment

elstehle commented Aug 25, 2022

Adds support for json lines format to the nested JSON reader #11534

Adds support for json lines format to the nested JSON reader #11534

Conversation

elstehle commented Aug 15, 2022 • edited Loading

Checklist

codecov bot commented Aug 15, 2022 • edited Loading

Codecov Report

elstehle commented Aug 24, 2022

galipremsagar left a comment

Choose a reason for hiding this comment

karthikeyann left a comment

Choose a reason for hiding this comment

karthikeyann Aug 25, 2022

Choose a reason for hiding this comment

karthikeyann Aug 25, 2022

Choose a reason for hiding this comment

elstehle Aug 25, 2022 • edited Loading

Choose a reason for hiding this comment

karthikeyann left a comment

Choose a reason for hiding this comment

elstehle commented Aug 25, 2022

elstehle commented Aug 15, 2022 •

edited

Loading

codecov bot commented Aug 15, 2022 •

edited

Loading

elstehle Aug 25, 2022 •

edited

Loading