arkouda_server fails on read_hdf when trying to access rank 4 datasets (corrupted size vs. prev_size) #702
Assignees
Labels
enhancement
New feature or request
Comments
|
Are you trying to use a multi-dimensional shape with Arkouda? We don't support that yet. Maybe this is missed in the error handling. @reuster986 @hokiegeek2 |
|
@wolfiex people have gotten around this buy rolling their own multi-dimensional wrapper class in python. |
|
Ah, that would be it!
Is there a good way to slice the dataset upon load, or would I have to read
it using h5py and then convert it for use with arkouda?
|
|
@pierce314159 @Ethan-DeBandi99 now that we have ArrayView should we revisit this? |
|
@mhmerrill I would think so. I will touch base with @pierce314159 and see if I can get this functioning properly. |
|
@wolfiex - would it be possible for you to share your dataset? I am working on making multi-dimensional hdf5 functional in arkouda. |
stress-tess
pushed a commit
that referenced
this issue
Jul 14, 2022
* Adds read/write support for multi-dimensional objects in HDF5. Supports read/write of objects as flattened arrays or multidim. Save/load functionality added to arrayview. This is an initial implementation that is stand alone and will require further integration with the rest of the HDF5 module." * Added FILEIO.md to detail the schema used for multidimensional HDF5. * Correcting mypy error. * Correcting circular import * Pulls array local to write it. This is not ideal, but this code is the first iteration for this. Adds updated error handling. * Updated to pull shape local before writing. * Addressing review comments. * addressing review comments * PR Review updates. * Updating module name to fit standard convention. * Formatting fix. * Adding semicolon * Rewording for Pierce.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Using the same file, I am able to list all the datasets and extract values from datasets with shape
Dataset {1/Inf, 101, 545}, however each time I try to do the same on a variable with shapeDataset {1/Inf, 47, 101, 545}the code fails with*** Error in `/users/xxxx/scratch/arkouda/arkouda_server': corrupted size vs. prev_size: 0x00007fe52806c4c0 ***Is there a reason this might be happening. I have increased the available memory to well over what is required, with no luck whatsoever.
The text was updated successfully, but these errors were encountered: