Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parquet sorted column ArrayIndexOutOfBoundsException #5608

Closed
devinrsmith opened this issue Jun 12, 2024 · 0 comments · Fixed by #5758
Closed

Parquet sorted column ArrayIndexOutOfBoundsException #5608

devinrsmith opened this issue Jun 12, 2024 · 0 comments · Fixed by #5758
Assignees
Labels
bug Something isn't working core Core development tasks parquet Related to the Parquet integration
Milestone

Comments

@devinrsmith
Copy link
Member

I'm unable to sort a specific string column in parquet that has been exported by BigQuery. The values all appear to be null. The data can be all written out though, and upon reading the new parquet file, things seem OK. So, potentially this is specific to some edge case either in our code or the specifics of the parquet file.

java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0
    at io.deephaven.chunk.ObjectChunk.get(ObjectChunk.java:77)
    at io.deephaven.engine.table.impl.sources.regioned.ColumnRegionChunkDictionary.getObject(ColumnRegionChunkDictionary.java:64)
    at io.deephaven.engine.table.impl.sources.regioned.ColumnRegionChunkDictionary.lambda$fillChunkAppend$1(ColumnRegionChunkDictionary.java:80)
    at io.deephaven.engine.rowset.RowSet.lambda$forAllRowKeys$0(RowSet.java:312)
    at io.deephaven.engine.rowset.impl.sortedranges.SortedRanges.forEachLong(SortedRanges.java:444)
    at io.deephaven.engine.rowset.impl.sortedranges.SortedRanges.ixForEachLong(SortedRanges.java:4615)
    at io.deephaven.engine.rowset.impl.WritableRowSetImpl.forEachRowKey(WritableRowSetImpl.java:339)
    at io.deephaven.engine.rowset.RowSet.forAllRowKeys(RowSet.java:311)
    at io.deephaven.engine.table.impl.sources.regioned.ColumnRegionChunkDictionary.fillChunkAppend(ColumnRegionChunkDictionary.java:80)
    at io.deephaven.engine.page.Page$WithDefaults.fillChunk(Page.java:70)
    at io.deephaven.engine.page.PageStore.fillChunk(PageStore.java:85)
    at io.deephaven.engine.page.PageStore.fillChunk(PageStore.java:85)
    at io.deephaven.engine.table.impl.sources.regioned.RegionedColumnSourceBase.fillChunk(RegionedColumnSourceBase.java:56)
    at io.deephaven.engine.table.impl.SortHelpers.makeAndFillValues(SortHelpers.java:788)
    at io.deephaven.engine.table.impl.SortHelpers.doChunkSortingOne(SortHelpers.java:513)
    at io.deephaven.engine.table.impl.SortHelpers.getSortMappingOne(SortHelpers.java:469)
    at io.deephaven.engine.table.impl.SortHelpers.getSortedKeys(SortHelpers.java:263)
    at io.deephaven.engine.table.impl.SortHelpers.getSortedKeys(SortHelpers.java:231)
    at io.deephaven.engine.table.impl.SortOperation.initialize(SortOperation.java:238)
    at io.deephaven.engine.table.impl.QueryTable.lambda$getResultNoMemo$80(QueryTable.java:3732)
    at io.deephaven.engine.table.impl.BaseTable.initializeWithSnapshot(BaseTable.java:1293)
    at io.deephaven.engine.table.impl.QueryTable.lambda$getResultNoMemo$81(QueryTable.java:3731)
    at io.deephaven.engine.table.impl.perf.QueryPerformanceRecorder.withNugget(QueryPerformanceRecorder.java:369)
    at io.deephaven.engine.table.impl.QueryTable.getResultNoMemo(QueryTable.java:3720)
    at io.deephaven.engine.table.impl.QueryTable.lambda$getResult$79(QueryTable.java:3714)
    at io.deephaven.engine.table.impl.QueryTable$MemoizedResult.getOrCompute(QueryTable.java:3674)
    at io.deephaven.engine.table.impl.QueryTable.memoizeResult(QueryTable.java:3643)
    at io.deephaven.engine.table.impl.QueryTable.getResult(QueryTable.java:3713)
    at io.deephaven.engine.table.impl.QueryTable.sort(QueryTable.java:2711)
    at io.deephaven.engine.table.impl.QueryTable.sort(QueryTable.java:100)
    at io.deephaven.api.TableOperationsDefaults.sort(TableOperationsDefaults.java:37)
    at io.deephaven.engine.table.impl.SortHelpers.doSymbolTableMapping(SortHelpers.java:368)
    at io.deephaven.engine.table.impl.SortHelpers.getSortedKeys(SortHelpers.java:261)
    at io.deephaven.engine.table.impl.SortHelpers.getSortedKeys(SortHelpers.java:231)
    at io.deephaven.engine.table.impl.SortOperation.initialize(SortOperation.java:238)
    at io.deephaven.engine.table.impl.QueryTable.lambda$getResultNoMemo$80(QueryTable.java:3732)
    at io.deephaven.engine.table.impl.BaseTable.initializeWithSnapshot(BaseTable.java:1293)
    at io.deephaven.engine.table.impl.QueryTable.lambda$getResultNoMemo$81(QueryTable.java:3731)
    at io.deephaven.engine.table.impl.perf.QueryPerformanceRecorder.withNugget(QueryPerformanceRecorder.java:369)
    at io.deephaven.engine.table.impl.QueryTable.getResultNoMemo(QueryTable.java:3720)
    at io.deephaven.engine.table.impl.QueryTable.lambda$getResult$79(QueryTable.java:3714)
    at io.deephaven.engine.table.impl.QueryTable$MemoizedResult.getOrCompute(QueryTable.java:3674)
    at io.deephaven.engine.table.impl.QueryTable.memoizeResult(QueryTable.java:3643)
    at io.deephaven.engine.table.impl.QueryTable.getResult(QueryTable.java:3713)
    at io.deephaven.engine.table.impl.QueryTable.sort(QueryTable.java:2711)
    at io.deephaven.engine.table.impl.QueryTable.sort(QueryTable.java:100)
    at io.deephaven.engine.table.impl.UncoalescedTable.sort(UncoalescedTable.java:415)
    at io.deephaven.engine.table.impl.UncoalescedTable.sort(UncoalescedTable.java:43)
    at io.deephaven.server.table.ops.SortTableGrpcImpl.create(SortTableGrpcImpl.java:104)
    at io.deephaven.server.table.ops.SortTableGrpcImpl.create(SortTableGrpcImpl.java:26)
    at io.deephaven.server.table.ops.TableServiceGrpcImpl$BatchExportBuilder.doExport(TableServiceGrpcImpl.java:757)
    at io.deephaven.server.session.SessionState$ExportObject.doExport(SessionState.java:995)

Will share file directly with those interested.

@devinrsmith devinrsmith added bug Something isn't working triage labels Jun 12, 2024
@devinrsmith devinrsmith added this to the 3. Triage milestone Jun 12, 2024
@rcaudy rcaudy added core Core development tasks parquet Related to the Parquet integration and removed triage labels Jun 12, 2024
@rcaudy rcaudy modified the milestones: 3. Triage, June 2024 Jun 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working core Core development tasks parquet Related to the Parquet integration
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants