Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Test rollups and trees #3248

Open
rcaudy opened this issue Dec 29, 2022 · 1 comment
Open

Test rollups and trees #3248

rcaudy opened this issue Dec 29, 2022 · 1 comment
Assignees
@rcaudy
Labels
2023_triagedNoMilestone core Core development tasks feature request New feature or request hierarchical_tables query engine
Milestone
Backlog

Comments

@rcaudy
Copy link
Member

rcaudy commented Dec 29, 2022

With the merge of #3243, we have a complete implementation for rollups and trees, along with a number of other features and improvements. Testing, however, remains incomplete, and unit testing focused on the newly-available features is negligible.

This ticket should cover:

  • Further integration testing of rollups
  • Integration testing of trees
  • Development of a server-side snapshot test tool that will snapshot a hierarchical table and produce a table containing the resulting chunk data
  • Unit testing for rollups and trees
  • Fixing bugs found in testing
@rcaudy rcaudy added feature request New feature or request query engine core Core development tasks hierarchical_tables labels Dec 29, 2022
@rcaudy rcaudy added this to the Dec 2022 milestone Dec 29, 2022
@rcaudy rcaudy self-assigned this Dec 29, 2022
@rcaudy rcaudy mentioned this issue Dec 29, 2022
Merged
This was referenced Jan 9, 2023
Merged
Merged
Merged
Merged
@rcaudy
Copy link
Member Author

rcaudy commented Jan 13, 2023

@niloc132 managed to get this error by rapidly collapsing and expanding rollup nodes:

2023-01-13T16:48:07.630Z | heduler-Concurrent-1 | ERROR | icalTableViewSubscription | Internal Error '1c0206a4-bba4-4564-bd0c-bfe6c1b90876' io.deephaven.engine.table.impl.hierarchical.HierarchicalTableSnapshotException: HierarchicalTable.snapshot error
        at io.deephaven.engine.table.impl.hierarchical.HierarchicalTableImpl.visitExpandedNode(HierarchicalTableImpl.java:1274)
        at io.deephaven.engine.table.impl.hierarchical.HierarchicalTableImpl.visitExpandedNode(HierarchicalTableImpl.java:1126)
        at io.deephaven.engine.table.impl.hierarchical.HierarchicalTableImpl.visitExpandedNode(HierarchicalTableImpl.java:1257)
        at io.deephaven.engine.table.impl.hierarchical.HierarchicalTableImpl.visitExpandedNode(HierarchicalTableImpl.java:1126)
        at io.deephaven.engine.table.impl.hierarchical.HierarchicalTableImpl.visitExpandedNode(HierarchicalTableImpl.java:1257)
        at io.deephaven.engine.table.impl.hierarchical.HierarchicalTableImpl.traverseExpansionsAndFillSnapshotChunks(HierarchicalTableImpl.java:1120)
        at io.deephaven.engine.table.impl.hierarchical.HierarchicalTableImpl.lambda$snapshotData$5(HierarchicalTableImpl.java:1086)
        at io.deephaven.engine.table.impl.remote.ConstructSnapshot.callDataSnapshotFunction(ConstructSnapshot.java:1193)
        at io.deephaven.engine.table.impl.remote.ConstructSnapshot.callDataSnapshotFunction(ConstructSnapshot.java:1035)
        at io.deephaven.engine.table.impl.hierarchical.HierarchicalTableImpl.snapshotData(HierarchicalTableImpl.java:1083)
        at io.deephaven.engine.table.impl.hierarchical.HierarchicalTableImpl.snapshot(HierarchicalTableImpl.java:937)
        at io.deephaven.engine.table.impl.hierarchical.RollupTableImpl.snapshot(RollupTableImpl.java:55)
        at io.deephaven.server.hierarchicaltable.HierarchicalTableViewSubscription.buildAndSendSnapshot(HierarchicalTableViewSubscription.java:313)
        at io.deephaven.server.hierarchicaltable.HierarchicalTableViewSubscription.process(HierarchicalTableViewSubscription.java:278)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at io.deephaven.server.runner.DeephavenApiServerModule$ThreadFactory.lambda$newThread$0(DeephavenApiServerModule.java:165)
        at java.lang.Thread.run(Thread.java:829)
caused by:
java.lang.ArrayIndexOutOfBoundsException: Index 3 out of bounds for length 2
        at io.deephaven.engine.table.impl.sources.DoubleArraySource.lambda$fillSparseChunk$1(DoubleArraySource.java:261)
        at io.deephaven.engine.rowset.RowSequence.lambda$forAllRowKeys$0(RowSequence.java:176)
        at io.deephaven.engine.rowset.impl.RowSequenceRowKeysChunkImpl.forEachRowKey(RowSequenceRowKeysChunkImpl.java:205)
        at io.deephaven.engine.rowset.RowSequence.forAllRowKeys(RowSequence.java:175)
        at io.deephaven.engine.table.impl.sources.DoubleArraySource.fillSparseChunk(DoubleArraySource.java:257)
        at io.deephaven.engine.table.impl.sources.ArrayBackedColumnSource.fillChunk(ArrayBackedColumnSource.java:531)
        at io.deephaven.engine.table.impl.DefaultChunkSource.getChunkByFilling(DefaultChunkSource.java:44)
        at io.deephaven.engine.table.impl.sources.ArrayBackedColumnSource.getChunk(ArrayBackedColumnSource.java:599)
        at io.deephaven.temp.c_8fad1f863d9f3b980400db2fcd2c8950156deaaeb81908494ad135820a2673d5v55_0.Formula.fillChunk(Formula.java:123)
        at io.deephaven.engine.table.impl.sources.ViewColumnSource.fillChunk(ViewColumnSource.java:219)
        at io.deephaven.engine.table.impl.sources.RedirectedColumnSource$FillContext.doOrderedFillAndPermute(RedirectedColumnSource.java:696)
        at io.deephaven.engine.table.impl.sources.RedirectedColumnSource.doFillChunk(RedirectedColumnSource.java:436)
        at io.deephaven.engine.table.impl.sources.RedirectedColumnSource.fillChunk(RedirectedColumnSource.java:409)
        at io.deephaven.engine.table.impl.hierarchical.HierarchicalTableImpl$NodeFillContext.fillNormalDestinations(HierarchicalTableImpl.java:1422)
        at io.deephaven.engine.table.impl.hierarchical.HierarchicalTableImpl.consumeRowsUntilNextExpansion(HierarchicalTableImpl.java:1320)
        at io.deephaven.engine.table.impl.hierarchical.HierarchicalTableImpl.visitExpandedNode(HierarchicalTableImpl.java:1250)
        at io.deephaven.engine.table.impl.hierarchical.HierarchicalTableImpl.visitExpandedNode(HierarchicalTableImpl.java:1126)
        at io.deephaven.engine.table.impl.hierarchical.HierarchicalTableImpl.visitExpandedNode(HierarchicalTableImpl.java:1257)
        at io.deephaven.engine.table.impl.hierarchical.HierarchicalTableImpl.visitExpandedNode(HierarchicalTableImpl.java:1126)
        at io.deephaven.engine.table.impl.hierarchical.HierarchicalTableImpl.visitExpandedNode(HierarchicalTableImpl.java:1257)
        at io.deephaven.engine.table.impl.hierarchical.HierarchicalTableImpl.traverseExpansionsAndFillSnapshotChunks(HierarchicalTableImpl.java:1120)
        at io.deephaven.engine.table.impl.hierarchical.HierarchicalTableImpl.lambda$snapshotData$5(HierarchicalTableImpl.java:1086)
        at io.deephaven.engine.table.impl.remote.ConstructSnapshot.callDataSnapshotFunction(ConstructSnapshot.java:1193)
        at io.deephaven.engine.table.impl.remote.ConstructSnapshot.callDataSnapshotFunction(ConstructSnapshot.java:1035)
        at io.deephaven.engine.table.impl.hierarchical.HierarchicalTableImpl.snapshotData(HierarchicalTableImpl.java:1083)
        at io.deephaven.engine.table.impl.hierarchical.HierarchicalTableImpl.snapshot(HierarchicalTableImpl.java:937)
        at io.deephaven.engine.table.impl.hierarchical.RollupTableImpl.snapshot(RollupTableImpl.java:55)

Test query was:

import io.deephaven.api.agg.Aggregation
import io.deephaven.api.agg.spec.AggSpec
rows = emptyTable(1_000_000).updateView("Group=i", "N=i%347", "M=i%29")
changes = timeTable("00:00:00.0001").view("Group=i%1_000_000", "LastModified=Timestamp", "Value = (i * Math.sin(i))%6977", "Weight = (i * Math.sin(i)) % 7151").lastBy("Group")
// changes = timeTable("00:00:10").view("Group=i%1_000_000", "LastModified=Timestamp", "Value = (i * Math.sin(i))%6977", "Weight = (i * Math.sin(i)) % 7151").lastBy("Group")
t = rows.join(changes, "Group")
def rollup = t.rollup([Aggregation.of(AggSpec.avg(), "Value"), Aggregation.of(AggSpec.sum(), "Weight")], true, "N", "M")
rec1 = rollup.makeNodeOperationsRecorder(io.deephaven.engine.table.hierarchical.RollupTable.NodeType.Aggregated)
rec1 = rec1.formatColumns("Value = Value > 0 ? NO_FORMATTING : ALICEBLUE")
rec2 = rollup.makeNodeOperationsRecorder(io.deephaven.engine.table.hierarchical.RollupTable.NodeType.Constituent)
rec2 = rec2.formatColumns("Weight = Weight > 0 ? NO_FORMATTING : RED")
rollup2 = rollup.withNodeOperations(rec1, rec2)
rollupA = t.rollup([Aggregation.of(AggSpec.avg(), "Value"), Aggregation.of(AggSpec.sum(), "Weight")], false, "N", "M")

This was referenced Jan 27, 2023
Merged
Merged
@pete-petey pete-petey modified the milestones: Dec 2022, Apr 2023 Apr 19, 2023
@rcaudy rcaudy modified the milestones: Apr 2023, June 2023 May 31, 2023
@pete-petey pete-petey modified the milestones: June 2023, August 2023 Jul 31, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rcaudy rcaudy

Labels
2023_triagedNoMilestone core Core development tasks feature request New feature or request hierarchical_tables query engine
Projects
None yet
Milestone
Backlog
Development

No branches or pull requests
2 participants
@rcaudy @pete-petey