Implement adaptive partitioning for FTE based on input #17024
Conversation
| // Skip for write nodes since writing partitioned data with small amount of nodes could cause | ||
| // memory related issues even when the amount of data is small. Additionally, skip for FTE mode since we | ||
| // are not using estimated partitionCount in FTE scheduler. | ||
| if (PlanNodeSearcher.searchFrom(plan).whereIsInstanceOfAny(INSERT_NODES).matches() | ||
| || getRetryPolicy(session) == RetryPolicy.TASK) { | ||
| // For streaming execution, skip for write nodes since writing partitioned data with small amount of nodes could cause | ||
| // memory related issues even when the amount of data is small. | ||
| if (!getRetryPolicy(session).equals(RetryPolicy.TASK) && PlanNodeSearcher.searchFrom(plan).whereIsInstanceOfAny(INSERT_NODES).matches()) { | ||
| return plan; | ||
| } |
There was a problem hiding this comment.
Adaptive partitioning is disabled for writes in streaming execution, is this a problem in FTE as well? @arhimondr ?
There was a problem hiding this comment.
Yes - it makes sense to keep lots of partitons if we are writing to avoid "too many open writers" problem. Not sure if we need upper bound always.
|
Also, do we need additional fte-specific configs of |
Technically we do not support mixed workloads - so maybe not. @arhimondr WDYT? |
linzebing
force-pushed
the
fte-adaptive-partitioning
branch
from
6200393 to
4ec3971
Compare
|
|
linzebing
force-pushed
the
fte-adaptive-partitioning
branch
from
4ec3971 to
fa54a3d
Compare
|
Failure is unrelated |
|
@losipiuk @arhimondr : do you think it makes sense to introduce a flag to enable adaptive partitioning for writing? It's meaningful in the fact that we can enable this flag by setting |
IDK how important it is in practice. But if sth I would rather allow for configuring different minimum for writer and non-writer queries. |
linzebing
force-pushed
the
fte-adaptive-partitioning
branch
from
fa54a3d to
6d80ea4
Compare
linzebing
force-pushed
the
fte-adaptive-partitioning
branch
from
6d80ea4 to
578a6c7
Compare
linzebing
changed the title
Description
Extend adaptive partitioning to fault-tolerant execution. Testing on tpcds-sf100 shows 4% improvement on wall time and 8% improvement on CPU time. Testing on tpcds-sf1000 shows 4.2% improvement on wall time and 1.9% improvement on CPU time.
Additional context and related issues
#15489
Release notes
(x) This is not user-visible or docs only and no release notes are required.
( ) Release notes are required, please propose a release note for me.
( ) Release notes are required, with the following suggested text: