[FEA] How to read hex as int using cudf

[VibhuJawa]

What is your question?
What is the cudf equivalent way to read hex as int columns using cudf.

Example of doing this in pandas.

import cudf
import pandas as pd

# create hex csv
fname = 'test.csv'
df = pd.DataFrame()
df['hex_col'] = ['9512c20b']*10
df.to_csv(fname,index=False)

## read_csv using pandas
from functools import partial
df = pd.read_csv(fname, converters={'hex_col': partial(int, base=16)})
df.head(5)

Output

	hex_col
0	2501034507
1	2501034507
2	2501034507
3	2501034507
4	2501034507

[mjsamoht]

If the numbers start with "0x" they will be interpreted as hexadecimal numbers. This was added in #973.

[VibhuJawa]

We have datasets where hex-numbers do not start with 0x . We want this feature for such numbers.

For eg: We have numbers like 9512c20b which are not appended with 0x . There was discussion about this in the rapidsdev-io channel.

[mjsamoht]

I see. That is currently not supported. One option could be to add a hex datatype that can be specified in dtype. Or we could perhaps automatically treat [0-9a-f] as "digits".

[VibhuJawa]

I see. That is currently not supported. One option could be to add a hex datatype that can be specified in dtype. Or we could perhaps automatically treat [0-9a-f] as "digits".

FWIW, I like the first option of adding a hex datatype.

Automatically treating [0-9a-f] as "digits" makes it inconsistent with pandas behavior .

[j-ieong]

@VibhuJawa For adding a hex dtype, do you expect it to be 32-bit or 64-bit for memory allocation? Or would you prefer more explicit hex32 and hex64?

If you have 9512c20b, then a simple print on the Python side would yield -1793932789 instead of 2501034507 that you may expect/get from PANDAS, as GDF_INT32 is a signed int underneath.

[VibhuJawa]

@VibhuJawa For adding a hex dtype, do you expect it to be 32-bit or 64-bit for memory allocation? Or would you prefer more explicit hex32 and hex64?

If you have 9512c20b, then a simple print on the Python side would yield -1793932789 instead of 2501034507 that you may expect/get from PANDAS, as GDF_INT32 is a signed int underneath.

I would prefer a more explicit hex32 and hex64 dtype.

I also do not sure know how to get , -1793932789 with print.

x = int("9512c20b", 16)
print(x)

Output

2501034507

@randerzander Thoughts ?

[j-ieong]

I would prefer a more explicit hex32 and hex64 dtype.

I also do not sure know how to get , -1793932789 with print.

x = int("9512c20b", 16)
print(x)

Output

2501034507

I believe int in Python 3 is larger than 32-bits. The output cudf dataframe would be returned as either GDF_INT32 or GDF_INT64 dtype; the PANDAS equivalent dataframe would be:

>>> import pandas as pd
>>> df = pd.DataFrame([int('9512c20b', 16)], dtype='int32')
>>> print(df)
            0
0 -1793932789
>>> df = pd.DataFrame([int('9512c20b', 16)], dtype='int64')
>>> print(df)
            0
0  2501034507

[VibhuJawa]

I would prefer a more explicit hex32 and hex64 dtype.
I also do not sure know how to get , -1793932789 with print.

x = int("9512c20b", 16)
print(x)

Output

2501034507

I believe int in Python 3 is larger than 32-bits. The output cudf dataframe would be returned as either GDF_INT32 or GDF_INT64 dtype; the PANDAS equivalent dataframe would be:

>>> import pandas as pd
>>> df = pd.DataFrame([int('9512c20b', 16)], dtype='int32')
>>> print(df)
            0
0 -1793932789
>>> df = pd.DataFrame([int('9512c20b', 16)], dtype='int64')
>>> print(df)
            0
0  2501034507

Aah, got it. I think with int32 we may have overflows (as in this case) . A lot of times the hex inputs are hashes created to anonymize the data.

We may even want to look at supporting bigger hex ints later.

import numpy as np

num = int('9512c20b', 16)
i32 = np.iinfo(np.int32)
i64 = np.iinfo(np.int64)
print(i32.max<num)
print(i64.max<num)
print(np.int32(num))
print(np.int64(num))

True
False
-1793932789
2501034507

[j-ieong]

So I decided on adding hex32, hex64, and simply hex. The latter aliases hex64 as its more user-friendly to default to the larger size, and hexadecimal data would probably require it.
If it's known that the dataset will not overflow, then hex32 is available to save some memory.

Let me know what you think...

[VibhuJawa]

So I decided on adding hex32, hex64, and simply hex. The latter aliases hex64 as its more user-friendly to default to the larger size, and hexadecimal data would probably require it.
If it's known that the dataset will not overflow, then hex32 is available to save some memory.

Let me know what you think...

I agree with this approach . I think alias will make it intuitive for the user . Thanks a lot for your work on this.