Fix table function execution without partitioning (v2) #21558
Conversation
findepi
force-pushed
the
findepi/exclude-columns-streaming-with-lifecycle
branch
from
5b36e66 to
a31bbfe
Compare
|
failure is related. |
|
I've tested the fix and it helps with #20398 however when writing the data using table function, it still does not distribute the data evenly and there is no way to control the number of files being written. |
between the nodes? there is no "writer scaling" equivalent for table functions, right?
do you mean control number of TF data processor instances? |
between files, when using
correct
control the number of files, for |
findepi
force-pushed
the
findepi/exclude-columns-streaming-with-lifecycle
branch
from
a31bbfe to
b358662
Compare
|
( rebased to resolve conflicts, no other changes ) |
This is indeed where the writers are created. But I guess this depends on the shape of the plan right before going into the table function operator? |
|
Planned changes moved out to #21710 |
findepi
force-pushed
the
findepi/exclude-columns-streaming-with-lifecycle
branch
3 times, most recently
from
e1bf212 to
d762983
Compare
|
This pull request has gone a while without any activity. Tagging the Trino developer relations team: @bitsondatadev @colebow @mosabua |
findepi
added
stale-ignore
findepi
force-pushed
the
findepi/exclude-columns-streaming-with-lifecycle
branch
from
d762983 to
fa9fdf0
Compare
During migration from TestNG to JUnit `@TestInstance(PER_CLASS)` annotation was added, but it implies single-threaded execution. Restore previous parallelism: either add `@Execution(CONCURRENT)` or inherit it from base class.
Previously, when table function did not declare partitioning, it would be globally distributed, but on a worker node it would run single-threaded and first buffer all data in memory, like a one big WINDOW. After the change, the local execution processes input pages in a streaming fashion. This commit also fixes property derivations for a case where table function is partitioned on empty list of symbols (global grouping).
findepi
force-pushed
the
findepi/exclude-columns-streaming-with-lifecycle
branch
from
fa9fdf0 to
a892820
Compare
github-actions
bot
added
iceberg
| pagesIndex, | ||
| 0, | ||
| pagesIndex.getPositionCount(), | ||
| new TableFunctionDataProcessor() |
There was a problem hiding this comment.
Do we need to wrap in a new anonymous class?
Locally, I directly passed the tableFunction which also resolved the TestJsonTable failures for me.
Previously, when table function did not declare partitioning it would run single-threaded and first buffer all data in memory, like a one big WINDOW. After the change, the local execution processes input pages in a streaming fashion.
Fixes #20398
Alternative to #21378, with different
TableFunctionDataProcessorlifecycle. The implementation creates one TableFunctionDataProcessor per operator for streaming processing.