Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Missing empty segments for parquet read of segarray with string values #2560

Closed
stress-tess opened this issue Jul 11, 2023 · 1 comment · Fixed by #2562
Closed

Missing empty segments for parquet read of segarray with string values #2560

stress-tess opened this issue Jul 11, 2023 · 1 comment · Fixed by #2562
Assignees
Labels
bug Something isn't working File IO Arkouda file IO capabilities

Comments

@stress-tess
Copy link
Member

Encountered while working #2539 and trying to extend testing of test_segarray_write to cover string values

In [5]: a, b, c = ["one", "two", "three"], ["un", "deux", "trois"], ["uno", "dos", "tres"]

In [6]: s = ak.SegArray(ak.array([0, 0, len(a), len(a), len(a), len(a) + len(c)]), ak.array(a + c))

In [7]: s
Out[7]:
SegArray([
[]
['one', 'two', 'three']
[]
[]
['uno', 'dos', 'tres']
[]
])

In [8]: s.to_parquet("segarray_test_empty")
Out[8]: 'Dataset written successfully!'

In [9]: rd_data = ak.read_parquet("segarray_test_empty*")

In [10]: rd_data
Out[10]:
SegArray([
['one', 'two', 'three']
['uno', 'dos', 'tres']
])
@stress-tess stress-tess added the File IO Arkouda file IO capabilities label Jul 11, 2023
@Ethan-DeBandi99 Ethan-DeBandi99 self-assigned this Jul 11, 2023
@Ethan-DeBandi99
Copy link
Contributor

Ethan-DeBandi99 commented Jul 11, 2023

@pierce314159 - is this with single locale or multiple? I definitely remember this working so I want to make sure I am not missing anything.

EDIT
I was able to reproduce this on single locale. I am also going to be testing on multi-locale, but will need to do so later.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working File IO Arkouda file IO capabilities
Projects
None yet
2 participants