You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When writing a parquet file with a boolean column I receive the error: EOF: unable to put boolean value.
To Reproduce
I don't yet have a minimal repro. For me I was writing batches of ~100k rows containing a uint64 column and a boolean column. The boolean column included null values.
Expected behavior
The boolean column to be correctly written.
Additional context
Based on debugging, I believe the problem may be the method used to extend the array:
When my case fails, values.len() is 27054. I suspect that even after adding 256 bytes there isn't enough room for all of the values (256 bytes is 2048 bits). It would actually need to be extended with 3382 bytes to have enough capacity for all of those rows.
It should probably be extended with something like min(256, values.len() / 8) or values.len() / 8 rounded up to a power of 2.
The text was updated successfully, but these errors were encountered:
Describe the bug
When writing a parquet file with a boolean column I receive the error:
EOF: unable to put boolean value
.To Reproduce
I don't yet have a minimal repro. For me I was writing batches of ~100k rows containing a uint64 column and a boolean column. The boolean column included null values.
Expected behavior
The boolean column to be correctly written.
Additional context
Based on debugging, I believe the problem may be the method used to extend the array:
When my case fails,
values.len()
is 27054. I suspect that even after adding 256 bytes there isn't enough room for all of the values (256 bytes is 2048 bits). It would actually need to be extended with 3382 bytes to have enough capacity for all of those rows.It should probably be extended with something like
min(256, values.len() / 8)
orvalues.len() / 8
rounded up to a power of 2.The text was updated successfully, but these errors were encountered: