Add documentation for adaptive partitioning and adaptive task sizing of FTE #17179
Conversation
| @@ -227,27 +227,17 @@ properties only apply to a ``TASK`` retry policy. | |||
| * - Property name | |||
| - Description | |||
| - Default value | |||
| * - ``fault-tolerant-execution-target-task-input-size`` | |||
There was a problem hiding this comment.
What's the behavior for users who still have this (and other removed properties) configured?
There was a problem hiding this comment.
server startup will fail. Let me add this property to DefunctConfigs
linzebing
force-pushed
the
adaptive-docs
branch
from
6d748e1 to
6be2866
Compare
| * - ``fault-tolerant-execution-arbitrary-distribution-compute-task-target-size-growth-factor`` | ||
| - Growth factor for adaptive sizing of non-writer tasks of arbitrary | ||
| distribution for fault-tolerant execution. | ||
| - ``1.2`` |
There was a problem hiding this comment.
Any upper or lower bounds? Also what does this mean .. explain "growth factor" ... I think it means you add 20% each time .. right?
There was a problem hiding this comment.
Lower bound is 1.0. Thanks for the reminder, I have added a commit to enforce this check.
think it means you add 20% each time .. right?
Yes. Let me add more color around this one.
linzebing
force-pushed
the
adaptive-docs
branch
from
6be2866 to
4238e1d
Compare
|
Failure is not related. |
linzebing
force-pushed
the
adaptive-docs
branch
from
4238e1d to
51d0dfe
Compare
| * **Default value:** ``false`` | ||
| * **Session property:** ``determine_partition_count_for_write_enabled`` | ||
|
|
||
| Enables determining the number of partitions based on amount of data read and processed by the |
There was a problem hiding this comment.
| Enables determining the number of partitions based on amount of data read and processed by the | |
| Enables determining the number of partitions based on the amount of data read and processed by the |
There was a problem hiding this comment.
This looks good now. Thank you @linzebing .. we will follow up with further improvements as discussed in a separate PR.
Description
Add documentation for adaptive partitioning and adaptive task sizing of FTE
Additional context and related issues
Add docs for #16719 and #17024
Release notes
(x) This is not user-visible or docs only and no release notes are required.
( ) Release notes are required, please propose a release note for me.
( ) Release notes are required, with the following suggested text: