[REVIEW] CSV Reader: Add hex dtype for explicit hexadecimal parsing
#2149
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Add `hex`, `hex64`, and `hex32` dtypes
- actual dtypes are `GDF_INT64`, `GDF_INT64`, and `GDF_INT32`
- indicates explicit request to parse with base-16
* Remove hexadecimal inference from dtype detection
* Combine column enablement, hex, and other flags
* Add pytests
j-ieong
changed the title
hex dtype for explicit hexadecimal parsinghex dtype for explicit hexadecimal parsing
OlivierNV
approved these changes
Jul 2, 2019
mjsamoht
approved these changes
Jul 2, 2019
j-ieong
added
3 - Ready for Review
thomcom
approved these changes
Jul 2, 2019
j-ieong
added
5 - Ready to Merge
mjsamoht
approved these changes
Jul 3, 2019
rapids-bot bot
pushed a commit
that referenced
this pull request
Mar 1, 2022
This PR silences warnings in `test_csv.py`. (I am working through one test file at a time so we can enable `-Werr` in the future.) The only warning in this file is related to integer overflow in pandas. Currently, the test data is as follows: https://github.com/rapidsai/cudf/blob/21325e8348f33b28e434d08d687a28f251c38f67/python/cudf/cudf/tests/test_csv.py#L1313-L1319 First, I note that this "hex" dtype is not part of the pandas API. It is a cuDF addition (#1925, #2149). Note that there are dtypes for `int32` / `hex32`, and the test data contains both a negative value `-0x1000` and a value `9512c20b`. The negative value `-0x1000` has a sensible interpretation if the results are meant to be signed, but then the value `9512c20b` is out of range (the maximum signed 32-bit value would be `0x7FFFFFFF` and the minimum signed 32-bit value would be `0x80000000`, using the big-endian convention of the parser). Recognizing this, pandas throws a `FutureWarning` when parsing the data `9512c20b` as `int32`, and unsafely wraps it to a negative value. This behavior will eventually be replaced by an `OverflowError`. In the future, we may need to decide if cuDF should raise an `OverflowError` when exceeding `0x7FFFFFFF` for consistency with pandas, or decide to use unsigned integers when parsing "hex" dtypes and compare to pandas' unsigned types in this test. Authors: - Bradley Dice (https://github.com/bdice) Approvers: - Vukasin Milovanovic (https://github.com/vuule) - https://github.com/brandon-b-miller URL: #10362
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #1925.
hex,hex64, andhex32dtypesGDF_INT64,GDF_INT64, andGDF_INT32determineBasefrom numeric parsing