pandas converts int32 to int64

[gdementen]

Is this intended? I had hoped no copying at all would happen in that case.

In [65]: a = np.random.randint(10, size=1e6)

In [66]: a.dtype
Out[66]: dtype('int32')

In [67]: b = np.random.randint(2, size=1e6)

In [68]: df = pandas.DataFrame({'a': a, 'b': b})

In [69]: df.dtypes
Out[69]:
a int64
b int64

[adamklein]

Yes, pandas has only four dtypes right now: int64, float64, bool, and object. This is in the interest of making it user-friendly, but at the expense of memory conservation obviously. In the future it might make sense to add more as long as it doesn't complicate the user-facing API.

[jseabold]

Just got bit by this, upcasting from float32, int8 and int16.

[cpcloud]

I actually like the fact that the dtypes are simpler when using pandas. Also, If you don't use a dict, then the dtype is preserved.

In practice is this a big deal? Maybe I'm a bit green, but I've never run into a situation using pandas where it really mattered whether I used int32 vs int64.

It matters for things like reading raw bytes from binary files, but if you're creating arrays large enough that the distinction between 32 and 64-bit width numbers matters, you'd be better off just getting more RAM.

For example, even if you had 4GB of RAM on your machine and you had a 2GB array of 32-bit integers, you're still going to need another 2GB if you want to do any non destructive arithmetic on that array thus maxing out your system's RAM.

Point is, doesn't seem like this is a bug. Just my two cents.

[wesm]

I agree that the simplicity is good-- you don't have to have to write down the dtype of a DataFrame like you do with a structured array. I think the design should be: have simple defaults, but when a data type is already set (e.g. int32), it's OK to "live and let live".

[adamsd5]

I am new to Pandas, but would like to put in my vote for supporting all ndarray types. From my testing, Series already will support other types, but DataFrame will not. I have two arguments... memory and speed. cpcloud suggested that you can always buy more memory, which is a reasonable suggestion. However, systems do have memory limits, and there are computation tasks that will use all of it (yes, even 256GB or more). Being able to fit twice as many samples on the system, regardless of how much memory you have, is a good thing.

On the speed front, I want to load binary files quickly into memory and process them with pandas. I wrote a C++ module for this purpose. I don't want to copy the memory after reading it from disk. For the processing we are doing, this would double the number of memory operations, which slows down the processing by almost 1/2. Unfortunately, after reading the binary into memory, I need to iterate over it and copy the int32 array into an int64 array. It is even worse than just a large memory copy because it also must up-cast each value to int64.

I like wesm's suggestion.

[cpcloud]

@adamsd5 You might try numpy's memmap ndarray subclass. It allows you to treat a file like an in-memory array. Of course, if your file is not just an array then this might be tricky. You could then pass the memmap to the pandas dataframe constructor and the dtype should be preserved. I agree with you that in the long run dtype preservation is desirable. Just out of curiosity, what kind of data are you working with?

[jreback]

@adamsd5 sounds like what you really want is out-of-core computation (similar to what @cpcloud suggested).
that is, your data is represented on disk, then slices are put in memory as needed and computed. I know @wesm has this as a goal as well. this will allow you to not even worry about the memory issue at all

HDFStore supports this now, though in a somewhat non-transparent manner.

Here's what you could do

1. store your data on-disk using HDFStore in a table format (could be a series of append operations from say csv files, or wherever you have now)
2. iterate over either a) a series of queries, or b) the indicies of the 'mapped frame'
3. compute and repeat

so imagine this pseudo code (this is the 2 b) part):

store = HDFStore('a_big_file.h5')

# pretend we have a store of the frame as a table 'df'

nrows = store.get_storer('df').nrows
chunk_size = 100000

for i in xrange(int(nrows / chunk_size) + 1):
    start_i = i * chunk_size
    stop_i = min((i + 1) * chunk_size, nrows)

    data_for_this_chunk = store.select('df', start = start_i, stop = stop_i)
    store.append('df_result', process_chunk(data_for_this_chunk))

would essentially give you a transformation operation, similar to process_chunk(df),
but processed in chunks. should be quite memory and speed insensitive, and could be easily parrallellizable
reduction operations are even simpler (as they can be accumulated in memory)

not that hard to create a wrapper around this....

[adamsd5]

cpcloud, does pandas.DataFrame treat such memmap ndarrays differently? You've presented a technique that I might use, but I think the DataFrame will still convert all int32 into int64.

I'm not actually trying to process things out of memory. I'm happy loading the entire Data Frame into memory. However, I would like to minimize the memory operations. Once the bytes are loaded from disk (and alas, I have no control over the format they are written), I do not want to copy them around at all (and I don't want pandas to make a copy for me either). From what I can tell, Pandas will always up-convert Int32 to Int64, which is a slow operation.

[cpcloud]

@adamsd5 A cursory glance at frame.py suggests that the dtype is preserved with instances of ndarray (isinstance tests for subclasses) that are not record arrays. You can also pass the dtype in the constructor.

[cpcloud]

@adamsd5 I was wrong. It seems that floating point types are preserved in the DataFrame constructor, but integer types are not. E.g.,

The issue still stands. I poked around in pandas/core/internals.py and saw that the function make_block converts any integer subtypes to int64, but preserves other types. Is there any reason to suspect that getting of the call to values.astype('i8') would break anything? Either way I'll try it and report back.

[jreback]

@cpcloud see PR #2705. this is a bit more complicated than it first appears, this change will appear on 0.10.2. the existing implementation will upcast for most operations, eg even though u can create a float32 (or int32) frame most operations will not preserve it

[jreback]

what dtypes should pandas fully support - this means all types of pad,fill,take,diff operations - their are specific cython functions created for each of the dtypes - the following are currently supported
float64,int64,int32,datetime64[ns],bool,object

float32 should be added clearly
what about float16,int16,int8,uint64,uint32,uint16,uint8?

you can always store these other dtypes, but certain operations will raise (or can auto upcast them)
eg say we don't support int16, can upcast to int32 and perform the ops

downside of adding more fully supported dtypes is additional compile time on installation and testing
and after a certain point prob should move to code generation (rather than copy/paste of the functions)
comments?

[adamsd5]

For my purposes, int32 and float32 would suffice. I see value in the smaller types for some people. If operations mean an upcast during the operation, the value is diminished. A use case would be a huge time series DataFrame on disk that has many int8 columns (perhaps factors), and the user wants to load, then filter based on time stamp and save a sub range of time. None of the int8 columns should be up converted. Just my ideas, hope it is helpful.

Darryl

On Jan 21, 2013, at 11:43 PM, jreback [email protected] wrote:

what dtypes should pandas fully support - this means all types of pad,fill,take,diff operations - their are specific cython functions created for each of the dtypes - the following are currently supported
float64,int64,int32,datetime64[ns],bool,object

float32 should be added clearly
what about float16,int16,int8,uint64,uint32,uint16,uint8?

you can always store these other dtypes, but certain operations will raise (or can auto upcast them)
eg say we don't support int16, can upcast to int32 and perform the ops

downside of adding more fully supported dtypes is additional compile time on installation and testing
and after a certain point prob should move to code generation (rather than copy/paste of the functions)
comments?

—
Reply to this email directly or view it on GitHub.

[wesm]

Boom. resolved by #2708, merged to master today