Expand CSV and JSON reader APIs to accept dtypes as a vector or map of data_type objects
#8856
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…fea-csv-dtypes-api
…fea-csv-dtypes-api
…fea-csv-dtypes-api
vuule
added
feature request
vuule
added
non-breaking
vuule
commented
Jul 26, 2021
| @@ -464,47 +475,71 @@ void reader::impl::set_column_names(device_span<uint64_t const> rec_starts, | |||
| } | |||
| } | |||
|
|
|||
| void reader::impl::set_data_types(device_span<uint64_t const> rec_starts, | |||
| rmm::cuda_stream_view stream) | |||
| std::vector<data_type> reader::impl::parse_data_types( | |||
There was a problem hiding this comment.
the diff is very confusing here. Did not rename the function, but added parse_data_types to handle the deprecated API that takes strings. This will make the future removal cleaner.
vuule
commented
Jul 26, 2021
Co-authored-by: Ram (Ramakrishna Prabhu) <[email protected]>
…to fea-csv-dtypes-api
galipremsagar
approved these changes
Aug 2, 2021
elstehle
reviewed
Aug 3, 2021
Co-authored-by: Elias Stehle <[email protected]>
rgsl888prabhu
approved these changes
Aug 4, 2021
ajschmidt8
approved these changes
Aug 4, 2021
vuule
added
5 - Ready to Merge
|
@gpucibot merge |
rapids-bot bot
pushed a commit
that referenced
this pull request
Aug 6, 2021
Found an compile error in `csv_test.cpp` that only occurs with a debug build since it exists in an `assert()` statement. Looks like this was introduced in PR #8856 Authors: - David Wendt (https://github.com/davidwendt) Approvers: - Vukasin Milovanovic (https://github.com/vuule) - Mike Wilson (https://github.com/hyperbolic2346) URL: #8981
shwina
pushed a commit
to shwina/cudf
that referenced
this pull request
Aug 9, 2021
Found an compile error in `csv_test.cpp` that only occurs with a debug build since it exists in an `assert()` statement. Looks like this was introduced in PR rapidsai#8856 Authors: - David Wendt (https://github.com/davidwendt) Approvers: - Vukasin Milovanovic (https://github.com/vuule) - Mike Wilson (https://github.com/hyperbolic2346) URL: rapidsai#8981
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Goal of the PR is to enable CSV to read columns as decimal, and to replace the string-based
dtypepart of the API.data_typebased API is needed because we need to specify scale for decimal columns, and doing this via a string that describes the type is 💩Changes in the PR:
dtyperelated getters/setters to also take a vector or a map ofdata_typeobjects.In case of CSV, vector of
data_types was already supported. Reworked the implementation to support different use cases that the "dtype-as-string" code path supports.parse_datesoption to make up for the special strings that CSV supported to denote that a column needs to be parsed as hexadecimal (the option to pass strings is to be removed).infer_dateoption toparse_dates.Breaking because API to specify date columns has been renamed to match the new
parse_hexAPI; renamed frominfer_datetoparse_datesDepends on #8843