Added streams to CSV reader and writer api #14340

shrshi · 2023-10-27T22:47:16Z

Description

This PR contributes to #13744.
-Added stream parameters to public APIs
cudf::io::read_csv
cudf::io::write_csv
-Added stream gtests

Checklist

I am familiar with the Contributing Guidelines.
New or existing tests cover these changes.
The documentation is up to date with these changes.

copy-pr-bot · 2023-10-27T22:47:19Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

shrshi · 2023-10-27T22:48:24Z

/ok to test

shrshi · 2023-10-31T05:40:25Z

/ok to test

…io-csv

karthikeyann · 2023-11-02T11:00:38Z

/ok to test

vuule

Nice and clean API change.
I think we can hit a few more code paths in the tests :)

vuule · 2023-11-08T22:46:43Z

cpp/tests/streams/io/csv_test.cpp

+TEST_F(CSVTest, CSVReader)
+{
+  constexpr auto num_rows = 10;
+  auto int8_values        = random_values<int8_t>(num_rows);


I think this the same situation as with ORC, where we don't really need the random values.

vuule · 2023-11-08T22:47:21Z

cpp/tests/streams/io/csv_test.cpp

+      .dtypes({cudf::data_type{cudf::type_id::INT8},
+               cudf::data_type{cudf::type_id::INT16},
+               cudf::data_type{cudf::type_id::INT32},
+               cudf::data_type{cudf::type_id::INT64},
+               cudf::data_type{cudf::type_id::UINT8},
+               cudf::data_type{cudf::type_id::UINT16},
+               cudf::data_type{cudf::type_id::UINT32},
+               cudf::data_type{cudf::type_id::UINT64},
+               cudf::data_type{cudf::type_id::FLOAT32},
+               cudf::data_type{cudf::type_id::FLOAT64}});


If we don't pass the data types the reader will launch an extra kernel to infer them, so we should not include this option :)

vuule · 2023-11-08T22:49:39Z

cpp/tests/streams/io/csv_test.cpp

+  auto int16_values       = random_values<int16_t>(num_rows);
+  auto int32_values       = random_values<int32_t>(num_rows);
+  auto int64_values       = random_values<int64_t>(num_rows);
+  auto uint8_values       = random_values<uint8_t>(num_rows);
+  auto uint16_values      = random_values<uint16_t>(num_rows);
+  auto uint32_values      = random_values<uint32_t>(num_rows);
+  auto uint64_values      = random_values<uint64_t>(num_rows);


AFAICT this can all be a single column, the code path is the same

vuule · 2023-11-08T22:50:55Z

cpp/tests/streams/io/csv_test.cpp

+  auto uint32_values      = random_values<uint32_t>(num_rows);
+  auto uint64_values      = random_values<uint64_t>(num_rows);
+  auto float32_values     = random_values<float>(num_rows);
+  auto float64_values     = random_values<double>(num_rows);


would be good to add a string column, there are additional thrust calls that post-process quotes (IIRC).

…io-csv

shrshi · 2023-11-13T20:48:50Z

/ok to test

shrshi · 2023-11-13T21:40:12Z

Thank you for the feedback, @vuule. Changing the tests did reveal more code paths creating string scalars using the default stream, which are now fixed.

cpp/src/io/csv/writer_impl.cu

shrshi · 2023-11-13T22:42:49Z

/ok to test

PointKernel · 2023-11-13T22:54:22Z

@shrshi you may want to resolve the merge conflicts in test CMake first to unblock CI.

shrshi · 2023-11-13T23:52:26Z

/ok to test

vuule

Thank you for tracking down the default stream use!
Couple more tiny changes

cpp/src/io/csv/writer_impl.cu

cpp/tests/streams/io/csv_test.cpp

…io-csv

shrshi · 2023-11-14T17:49:55Z

/ok to test

shrshi · 2023-11-14T20:03:39Z

/ok to test

ttnghia · 2023-11-14T22:27:30Z

cpp/include/cudf_test/column_wrapper.hpp

+    wrapped = cudf::make_strings_column(
+      d_chars, d_offsets, d_bitmask, null_count, cudf::test::get_default_stream());


Does this include changes from other PR? The additional stream parameter here and below seems unrelated to this PR.

My guess is that CSV stream test is exercising this code path while no prior stream test was, so before this PR we didn't observe that there was a missing stream argument here.

Yep, this change is needed for the string column in the tests - make_strings_column is called with the default stream otherwise.

PointKernel

LGTM

Thanks!

shrshi · 2023-11-14T22:39:02Z

/merge

Added streams to CSV reader and writer api

aa28cd8

github-actions bot added libcudf Affects libcudf (C++/CUDA) code. CMake CMake build issue labels Oct 27, 2023

shrshi added non-breaking Non-breaking change improvement Improvement / enhancement to an existing function labels Oct 27, 2023

shrshi marked this pull request as ready for review October 30, 2023 15:44

shrshi requested a review from a team as a code owner October 30, 2023 15:44

shrshi requested review from robertmaynard and karthikeyann October 30, 2023 15:44

Merge branch 'branch-23.12' into streams-io-csv

147c4dc

shrshi added 3 commits November 1, 2023 00:55

Merge branch 'branch-23.12' into streams-io-csv

15a8fb4

Merge branch 'streams-io-csv' of github.com:shrshi/cudf into streams-…

5b39b59

…io-csv

Merge branch 'branch-23.12' into streams-io-csv

102ee92

GregoryKimball assigned shrshi Nov 6, 2023

karthikeyann approved these changes Nov 8, 2023

View reviewed changes

vuule requested changes Nov 8, 2023

View reviewed changes

shrshi added 2 commits November 13, 2023 20:46

simplified tests

75076fd

Merge branch 'streams-io-csv' of github.com:shrshi/cudf into streams-…

c09d78f

…io-csv

shrshi requested a review from vuule November 13, 2023 20:52

PointKernel reviewed Nov 13, 2023

View reviewed changes

cpp/src/io/csv/writer_impl.cu Show resolved Hide resolved

cpp/src/io/csv/writer_impl.cu Outdated Show resolved Hide resolved

cpp/src/io/csv/writer_impl.cu Outdated Show resolved Hide resolved

code comment cleanup; initialization style

656a5b6

Merge branch 'branch-23.12' into streams-io-csv

c5011c0

vuule requested changes Nov 14, 2023

View reviewed changes

cpp/src/io/csv/writer_impl.cu Outdated Show resolved Hide resolved

cpp/src/io/csv/writer_impl.cu Outdated Show resolved Hide resolved

cpp/tests/streams/io/csv_test.cpp Outdated Show resolved Hide resolved

shrshi added 2 commits November 14, 2023 17:46

default mr usage; test header cleanup

08ac4a2

Merge branch 'streams-io-csv' of github.com:shrshi/cudf into streams-…

751936d

…io-csv

vuule approved these changes Nov 14, 2023

View reviewed changes

Merge branch 'branch-23.12' into streams-io-csv

413bfb6

ttnghia reviewed Nov 14, 2023

View reviewed changes

PointKernel approved these changes Nov 14, 2023

View reviewed changes

rapids-bot bot merged commit 27b052d into rapidsai:branch-23.12 Nov 14, 2023
62 of 63 checks passed

GregoryKimball mentioned this pull request Nov 22, 2023

[FEA] Expose public stream-ordered C++ APIs #13744

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added streams to CSV reader and writer api #14340

Added streams to CSV reader and writer api #14340

shrshi commented Oct 27, 2023

copy-pr-bot bot commented Oct 27, 2023

shrshi commented Oct 27, 2023

shrshi commented Oct 31, 2023

karthikeyann commented Nov 2, 2023

vuule left a comment

vuule Nov 8, 2023

vuule Nov 8, 2023

vuule Nov 8, 2023

vuule Nov 8, 2023

shrshi commented Nov 13, 2023

shrshi commented Nov 13, 2023

shrshi commented Nov 13, 2023

PointKernel commented Nov 13, 2023

shrshi commented Nov 13, 2023

vuule left a comment

shrshi commented Nov 14, 2023

shrshi commented Nov 14, 2023

ttnghia Nov 14, 2023

vyasr Nov 14, 2023

shrshi Nov 14, 2023

PointKernel left a comment

shrshi commented Nov 14, 2023

		wrapped = cudf::make_strings_column(
		d_chars, d_offsets, d_bitmask, null_count, cudf::test::get_default_stream());

Added streams to CSV reader and writer api #14340

Added streams to CSV reader and writer api #14340

Conversation

shrshi commented Oct 27, 2023

Description

Checklist

copy-pr-bot bot commented Oct 27, 2023

shrshi commented Oct 27, 2023

shrshi commented Oct 31, 2023

karthikeyann commented Nov 2, 2023

vuule left a comment

Choose a reason for hiding this comment

vuule Nov 8, 2023

Choose a reason for hiding this comment

vuule Nov 8, 2023

Choose a reason for hiding this comment

vuule Nov 8, 2023

Choose a reason for hiding this comment

vuule Nov 8, 2023

Choose a reason for hiding this comment

shrshi commented Nov 13, 2023

shrshi commented Nov 13, 2023

shrshi commented Nov 13, 2023

PointKernel commented Nov 13, 2023

shrshi commented Nov 13, 2023

vuule left a comment

Choose a reason for hiding this comment

shrshi commented Nov 14, 2023

shrshi commented Nov 14, 2023

ttnghia Nov 14, 2023

Choose a reason for hiding this comment

vyasr Nov 14, 2023

Choose a reason for hiding this comment

shrshi Nov 14, 2023

Choose a reason for hiding this comment

PointKernel left a comment

Choose a reason for hiding this comment

shrshi commented Nov 14, 2023