Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement get_json_object_multiple_paths #2233

Merged
merged 43 commits into from
Jul 18, 2024
Merged

Implement get_json_object_multiple_paths #2233

merged 43 commits into from
Jul 18, 2024

Conversation

ttnghia
Copy link
Collaborator

@ttnghia ttnghia commented Jul 17, 2024 •
edited
Loading

This re-implements get_json_object to support extracting multiple JSON objects from the same JSON input. By processing multiple JSON paths in just one kernel call, we can improve the performance of SQL queries in which there are large numbers of calls to get_json_object.

The building blocks of this work are very simple:

  • Employ an entire warp of threads (32 threads) to process only one row. This eliminates thread divergence and avoids threads from blocking each other due to executing different code paths on different input.
  • Enhance memory access by using consecutive warps to process the same input rows, just different JSON paths.

Depends on:

Benchmark on a dataset of 200'000 rows, given by a 6.7GB parquet file, and 80 JSON paths. The data is read using parquet chunked reader with 256MB output limit.

Baseline (before this work): 410 seconds.
With this work: 44 seconds.

ttnghia added 19 commits July 2, 2024 21:40
@ttnghia
Implement interface for getJsonObjectMultiplePaths
5277dd4 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Rewrite json_generator
8239ad5
@ttnghia
Merge branch 'branch-24.08' into get_json_object_long_strings
495d261
@ttnghia
Debug print
5b4c98f
@ttnghia
Cleanup header
b0423a7
@ttnghia
Add const and static qualifiers
b457d01
@ttnghia
Cleanup
33a17ee 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Faster name concatenation
fe1b678 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Add case path enum
7dd3cd2 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Fix try_skip_children
b43b971 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Implement get_json_object_multiple_paths
92eeb68 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Construct path command async
6606b9e
@ttnghia
More cleanup
767f50f 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Add comment
9ef9249 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Fix format
f41c3af 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Merge branch 'cleanup_get_json_object' into get_json_object_long_strings
c34ad25 
# Conflicts:
#	src/main/cpp/src/get_json_object.cu
@ttnghia
Construct path instructions for all paths at once
64af594 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Cleanup
bddbae7 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Adjust kernel launch parameters
55f00e4 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia ttnghia self-assigned this Jul 17, 2024
@ttnghia ttnghia mentioned this pull request Jul 17, 2024
Merged
@ttnghia ttnghia force-pushed the get_json_object_long_strings branch from 8814752 to e4d1352 Compare July 17, 2024 07:02
@ttnghia
Move function
750fcca 
Signed-off-by: Nghia Truong <[email protected]>
ttnghia added 2 commits July 17, 2024 10:16
@ttnghia
Handle all nulls input
ef6d7c2 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Merge branch 'branch-24.08' into get_json_object_long_strings
a7f8960 
# Conflicts:
#	src/main/cpp/src/get_json_object.cu
ttnghia added 4 commits July 17, 2024 14:17
@ttnghia
Change threshold
4eada79 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Fix style
7060016 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Change benchmark
1bf678d 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Merge branch 'get_json_object_multiple_paths_interface' into get_json…
79594ff 
…_object_long_strings

# Conflicts:
#	src/main/cpp/src/get_json_object.hpp
@ttnghia ttnghia mentioned this pull request Jul 17, 2024
Closed
@ttnghia
Copy link
Collaborator Author

ttnghia commented Jul 18, 2024

build

ttnghia added 2 commits July 17, 2024 19:54
@ttnghia
Copy link
Collaborator Author

ttnghia commented Jul 18, 2024

build

@ttnghia ttnghia marked this pull request as ready for review July 18, 2024 05:40
@ttnghia ttnghia requested review from thirtiseven and res-life July 18, 2024 05:41
@ttnghia
Fix num_blocks computation
dff238a 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Copy link
Collaborator Author

ttnghia commented Jul 18, 2024

build

@ttnghia
Fix move
508e0fe 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Fix TODO
7389ec0 
Signed-off-by: Nghia Truong <[email protected]>
@res-life
Copy link
Collaborator

LGTM.

@res-life
Copy link
Collaborator

We may add a case for the out-of-bound write scenario for multipile paths in a follow-up PR.

@ttnghia
Copy link
Collaborator Author

ttnghia commented Jul 18, 2024

build

@ttnghia ttnghia merged commit 05b05a0 into NVIDIA:branch-24.08 Jul 18, 2024
3 checks passed
@ttnghia ttnghia deleted the get_json_object_long_strings branch July 18, 2024 16:58
@ttnghia ttnghia mentioned this pull request Aug 14, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@res-life res-life res-life approved these changes

@thirtiseven thirtiseven Awaiting requested review from thirtiseven

Assignees

@ttnghia ttnghia

Labels
performance
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@ttnghia @res-life @sameerz