YARN-11073. Avoid unnecessary preemption for tiny queues under certain corner cases #4110
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
aajisaka
approved these changes
Mar 28, 2022
| * @param queues | ||
| * the list of queues to consider | ||
| * @param ignoreGuar | ||
| * ignore guarantee. | ||
| */ | ||
| private void resetCapacity(Resource clusterResource, | ||
| Collection<TempQueuePerPartition> queues, boolean ignoreGuar) { | ||
| private void resetCapacity(Collection<TempQueuePerPartition> queues, |
There was a problem hiding this comment.
For testing, I think we can make this method static package-private and add @VisibleForTesting annotation. That way we can call this method directly from test class. Note that the package of the annotation must be "org.apache.hadoop.thirdparty.com.google.common.annotations".
aajisaka
changed the title
HarshitGupta11
pushed a commit
to HarshitGupta11/hadoop
that referenced
this pull request
Nov 28, 2022
…n corner cases (apache#4110) Co-authored-by: Jian Chen <[email protected]> Signed-off-by: Akira Ajisaka <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of PR (YARN-11073)
When running a Hive job in a low-capacity queue on an idle cluster, preemption kicked in to preempt job containers even though there's no other job running and competing for resources.
Let's take this scenario as an example:
During the fifo preemption candidates selection process, the preemptableAmountCalculator needs to first "computeIdealAllocation" which depends on each queue's guaranteed/min capacity. A queue's guaranteed capacity is currently calculated as "Resources.multiply(totalPartitionResource, absCapacity)", so the guaranteed capacity of queue_low is:
queue_low: <Memory: (168*0.01)GB, VCores:(48*0.01)> = <Memory:1.68GB, VCores:0.48>, but since the Resource object takes only Long values, these Doubles values get casted into Long, and then the final result becomes<Memory:1GB, VCores:0>Because the guaranteed capacity of queue_low is 0, its normalized guaranteed capacity based on active queues is also 0 based on the current algorithm in "resetCapacity". This eventually leads to the continuous preemption of job containers running in queue_low.
In order to work around this corner case, "resetCapacity" needs to consider a couple new scenarios:
if the sum of absoluteCapacity/minCapacity of all active queues is zero, we should normalize their guaranteed capacity evenly:
1.0f / num_of_queuesif the sum of pre-normalized guaranteed capacity values (MB or VCores) of all active queues is zero, meaning we might have several queues like queue_low whose capacity value got casted into 0, we should normalize evenly as well like the first scenario (if they are all tiny, it really makes no big difference, for example, 1% vs 1.2%).
if one of the active queues has a zero pre-normalized guaranteed capacity value but its absoluteCapacity/minCapacity is not zero, then we should normalize based on the weight of their configured queue absoluteCapacity/minCapacity. This is to make sure queue_low gets a small but fair normalized value when queue_mid is also active.
minCapacity / (sum_of_min_capacity_of_active_queues)How was this patch tested?
The patch was tested on a small cluster with queues configured to be same as the scenario described above, verified that
For code changes:
LICENSE,LICENSE-binary,NOTICE-binaryfiles?@aajisaka