Add column_device_view pointers to EncColumnDesc #7097

kaatish · 2021-01-07T19:28:37Z

Closes #6893, closes #6894. Contributes to #5682

Reduce usage of stats_column_desc members in Parquet writer with column_device_view members.

…arquet-writer-col-device-view

devavret

Going great so far!

cpp/src/io/parquet/page_dict.cu

cpp/src/io/parquet/page_enc.cu

cpp/src/io/parquet/parquet_gpu.hpp

cpp/src/io/parquet/writer_impl.cu

cpp/src/io/parquet/page_enc.cu

cpp/src/io/parquet/writer_impl.cu

…arquet-writer-col-device-view

Avoid early exit for empty table

cpp/src/io/parquet/parquet_gpu.hpp

cpp/src/io/parquet/writer_impl.cu

codecov · 2021-01-21T21:31:43Z

Codecov Report

Merging #7097 (3d80045) into branch-0.18 (8860baf) will increase coverage by 0.11%.
The diff coverage is n/a.

@@               Coverage Diff               @@
##           branch-0.18    #7097      +/-   ##
===============================================
+ Coverage        82.09%   82.20%   +0.11%     
===============================================
  Files               97      100       +3     
  Lines            16474    16952     +478     
===============================================
+ Hits             13524    13936     +412     
- Misses            2950     3016      +66

Impacted Files	Coverage Δ
python/cudf/cudf/__init__.py	`100.00% <ø> (ø)`
python/cudf/cudf/_fuzz_testing/avro.py	`0.00% <ø> (ø)`
python/cudf/cudf/_fuzz_testing/csv.py	`0.00% <ø> (ø)`
python/cudf/cudf/_fuzz_testing/fuzzer.py	`0.00% <ø> (ø)`
python/cudf/cudf/_fuzz_testing/io.py	`0.00% <ø> (ø)`
python/cudf/cudf/_fuzz_testing/json.py	`0.00% <ø> (ø)`
python/cudf/cudf/_fuzz_testing/main.py	`0.00% <ø> (ø)`
python/cudf/cudf/_fuzz_testing/orc.py	`0.00% <ø> (ø)`
python/cudf/cudf/_fuzz_testing/parquet.py	`0.00% <ø> (ø)`
python/cudf/cudf/_fuzz_testing/utils.py	`0.00% <ø> (ø)`
... and 83 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fe5e07d...fa5858d. Read the comment docs.

devavret · 2021-01-21T21:58:31Z

You should be able to remove stringdata_to_nvstrdesc() and the code around its call site now.

cpp/src/io/parquet/writer_impl.cu

vuule

partial review.
Gotta say, static_cast -> element and strdesc mess -> string_view look 🔥

cpp/src/io/parquet/page_dict.cu

cpp/src/io/parquet/page_enc.cu

davidwendt

Looks like you may need to add some parquet tests that use strings columns that contain non-ASCII UTF-8 characters.

cpp/src/io/parquet/page_enc.cu

cpp/src/io/parquet/page_dict.cu

cpp/src/io/parquet/page_enc.cu

…arquet-writer-col-device-view

cpp/include/cudf/column/column_device_view.cuh

vuule

Looks good, couple of small changes.

cpp/src/io/parquet/page_enc.cu

vuule · 2021-01-29T21:05:36Z

cpp/src/io/parquet/page_enc.cu

+ * @param[out] col_desc Column description array [column_id]
+ * @param[out] leaf_column_views Device vector to store leaf columns


If possible, prefer return value to output parameters.

cpp/src/io/parquet/parquet_gpu.hpp

cpp/src/io/parquet/writer_impl.cu

vuule · 2021-02-02T00:49:02Z

rerun tests

harrism · 2021-02-02T22:47:54Z

@kaatish @vuule is this ready to merge assuming we can get CI / style to pass? Please update labels if so.

vuule · 2021-02-02T22:51:18Z

@kaatish @vuule is this ready to merge assuming we can get CI / style to pass? Please update labels if so.

I think we need @devavret 's review to merge.

devavret

Just one tiny thing but I've already changed it in my branch so can be let go.

cpp/src/io/parquet/page_enc.cu

devavret · 2021-02-03T19:56:46Z

rerun tests

vuule · 2021-02-04T07:47:39Z

rerun tests

vuule · 2021-02-04T16:52:14Z

@gpucibot merge

Add column_device_view pointers to EncColumnDesc

26f96bf

kaatish added code quality libcudf Affects libcudf (C++/CUDA) code. cuIO cuIO issue labels Jan 7, 2021

kaatish requested a review from devavret January 7, 2021 19:28

kaatish added 5 commits January 8, 2021 12:50

Merge branch 'branch-0.18' of https://github.com/rapidsai/cudf into p…

1f5b6c4

…arquet-writer-col-device-view

Fix compilation

9aba2b5

Merge branch 'branch-0.18' of https://github.com/rapidsai/cudf into p…

1946fed

…arquet-writer-col-device-view

Replace outdated calls in page_dict

d507132

Add GetDtypeLogicalLen for column_device_view

d286962

devavret requested changes Jan 12, 2021

View reviewed changes

kaatish added 5 commits January 14, 2021 12:41

PR comment fixes

f52fa95

Merge branch 'branch-0.18' of https://github.com/rapidsai/cudf into p…

bc4c771

…arquet-writer-col-device-view

Merge branch 'branch-0.18' of https://github.com/rapidsai/cudf into p…

a5064bb

…arquet-writer-col-device-view

Merge branch 'branch-0.18' of https://github.com/rapidsai/cudf into p…

e5d02f2

…arquet-writer-col-device-view

Fix tests

d19b8f1

kaatish requested a review from devavret January 21, 2021 04:03

kaatish added 2 - In Progress Currently a work in progress non-breaking Non-breaking change bug Something isn't working labels Jan 21, 2021

Schema fix

f7658c6

Avoid early exit for empty table

devavret requested changes Jan 21, 2021

View reviewed changes

cpp/src/io/parquet/parquet_gpu.hpp Outdated Show resolved Hide resolved

cpp/src/io/parquet/parquet_gpu.hpp Show resolved Hide resolved

cpp/src/io/parquet/writer_impl.cu Outdated Show resolved Hide resolved

devavret reviewed Jan 21, 2021

View reviewed changes

cpp/src/io/parquet/writer_impl.cu Outdated Show resolved Hide resolved

PR review fixes

90b5da0

kaatish requested a review from devavret January 22, 2021 18:21

kaatish marked this pull request as ready for review January 22, 2021 20:33

devavret reviewed Jan 25, 2021

View reviewed changes

cpp/src/io/parquet/writer_impl.cu Outdated Show resolved Hide resolved

vuule requested changes Jan 28, 2021

View reviewed changes

davidwendt requested changes Jan 28, 2021

View reviewed changes

cpp/src/io/parquet/page_enc.cu Outdated Show resolved Hide resolved

cpp/src/io/parquet/page_enc.cu Outdated Show resolved Hide resolved

cpp/src/io/parquet/page_dict.cu Outdated Show resolved Hide resolved

cpp/src/io/parquet/page_enc.cu Outdated Show resolved Hide resolved

kaatish added 6 commits January 28, 2021 10:18

Merge branch 'branch-0.18' of https://github.com/rapidsai/cudf into p…

d0e2613

…arquet-writer-col-device-view

PR review fixes

57d6afd

Merge branch 'branch-0.18' of https://github.com/rapidsai/cudf into p…

c2393a8

…arquet-writer-col-device-view

PR review fixes

f5203f5

Merge branch 'branch-0.18' of https://github.com/rapidsai/cudf into p…

9b2070b

…arquet-writer-col-device-view

Format fixes

25301a9

kaatish requested review from devavret, vuule and davidwendt January 29, 2021 19:47

davidwendt reviewed Jan 29, 2021

View reviewed changes

cpp/include/cudf/column/column_device_view.cuh Outdated Show resolved Hide resolved

Removed get_mask_offset_word

3522264

davidwendt approved these changes Jan 29, 2021

View reviewed changes

vuule requested changes Jan 29, 2021

View reviewed changes

PR review fixes

bd0691f

kaatish requested a review from vuule February 1, 2021 20:53

vuule approved these changes Feb 2, 2021

View reviewed changes

devavret approved these changes Feb 2, 2021

View reviewed changes

cpp/src/io/parquet/page_enc.cu Outdated Show resolved Hide resolved

PR review fix

fa5858d

rapids-bot bot merged commit 8334700 into rapidsai:branch-0.18 Feb 4, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add column_device_view pointers to EncColumnDesc #7097

Add column_device_view pointers to EncColumnDesc #7097

kaatish commented Jan 7, 2021 •

edited by vuule

Loading

devavret left a comment

codecov bot commented Jan 21, 2021 •

edited

Loading

devavret commented Jan 21, 2021

vuule left a comment

davidwendt left a comment

vuule left a comment

vuule Jan 29, 2021

vuule commented Feb 2, 2021

harrism commented Feb 2, 2021

vuule commented Feb 2, 2021

devavret left a comment

devavret commented Feb 3, 2021

vuule commented Feb 4, 2021

vuule commented Feb 4, 2021

		* @param[out] col_desc Column description array [column_id]
		* @param[out] leaf_column_views Device vector to store leaf columns

Add column_device_view pointers to EncColumnDesc #7097

Add column_device_view pointers to EncColumnDesc #7097

Conversation

kaatish commented Jan 7, 2021 • edited by vuule Loading

devavret left a comment

Choose a reason for hiding this comment

codecov bot commented Jan 21, 2021 • edited Loading

Codecov Report

devavret commented Jan 21, 2021

vuule left a comment

Choose a reason for hiding this comment

davidwendt left a comment

Choose a reason for hiding this comment

vuule left a comment

Choose a reason for hiding this comment

vuule Jan 29, 2021

Choose a reason for hiding this comment

vuule commented Feb 2, 2021

harrism commented Feb 2, 2021

vuule commented Feb 2, 2021

devavret left a comment

Choose a reason for hiding this comment

devavret commented Feb 3, 2021

vuule commented Feb 4, 2021

vuule commented Feb 4, 2021

kaatish commented Jan 7, 2021 •

edited by vuule

Loading

codecov bot commented Jan 21, 2021 •

edited

Loading