Decrease minCapacity of instances to 27
#9369
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Size Change: 0 B Total Size: 697 kB ℹ️ View Unchanged
|
ioannakok
force-pushed
the
mob/reduce-min-instances
branch
from
1367910 to
648d1cd
Compare
|
Any thoughts from @guardian/devx-reliability on this? |
I agree that it makes sense to experiment with the minimum capacity here 👍 IIUC we are unsure exactly how low we can set the minimum without impacting users? However, we know that the cost of deploying this type of change is very low, so I would be tempted to drop the capacity in groups of 31 and monitor at each stage using the metrics that you've suggested (stopping somewhere before we get to 15 where there are known problems!). I think it'd also be worth documenting the cost saving per month that we achieve for each capacity reduction to help to prioritise how far we want to push this experimentation. Footnotes
|
ioannakok
added a commit
that referenced
this pull request
Nov 1, 2023
@jacobwinch made a good point in the review about dropping the capacity in groups of 3 and monitoring at each stage. It's good to modify ASG capacity in groups of 3 so that AWS can evenly distribute across Availability Zones (https://docs.aws.amazon.com/autoscaling/ec2/userguide/auto-scaling-benefits.html#arch-AutoScalingMultiAZ). #9369 (comment) Co-authored-by: Jacob Winch <[email protected]>
`minCapacity` was bumped from 15 to 30 in #8724. After the fronts migration was completed, it seemed that 15 instances weren't holding up with the new traffic we were sending. That was a hotfix and there is upcoming work to split our stacks (#8351) but until we reach this point we would like to see whether we could handle the traffic with 25 instances. This change is part of the research we're doing to optimise how we scale: #9322. Co-authored-by: George B <[email protected]> Co-authored-by: Ravi <[email protected]>
@jacobwinch made a good point in the review about dropping the capacity in groups of 3 and monitoring at each stage. It's good to modify ASG capacity in groups of 3 so that AWS can evenly distribute across Availability Zones (https://docs.aws.amazon.com/autoscaling/ec2/userguide/auto-scaling-benefits.html#arch-AutoScalingMultiAZ). #9369 (comment) Co-authored-by: Jacob Winch <[email protected]>
ioannakok
force-pushed
the
mob/reduce-min-instances
branch
from
7c20146 to
5eb07b1
Compare
ioannakok
changed the title
minCapacity of instances to 25minCapacity of instances to 27
Thanks for the review @jacobwinch! That all makes perfect sense 👍 I have set |
|
Deployed at 12:41. ASG has reduced desired and minimum.
|
Closed
|
After this change we've noticed:
We will check again at the end of the month and compare with October to have a better view of the impact. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this change?
Reduces DCR
minCapacityto 27.Why?
minCapacitywas bumped from 15 to 30 inAfter the fronts migration was completed, it seemed that 15 instances weren't holding up with the new traffic. Bumping instances to 30 was a hotfix and there is upcoming work to split our stacks. Until we reach this point we would like to see whether we could handle the traffic with 27 instances. This change is a first step of the research we're doing to how to optimise DCR scaling in terms of:
See WIP document on our research and findings so far.
Metrics to keep an eye on
After the fronts migration and with DCR having a
minCapacityof 15 @guardian/devx-reliability team had to lower the SLO target forfaciaapp in https://github.com/guardian/slo-alerts/pull/37 after ongoing issues with DCR latency. After increasingminCapacityto 30 they were able to restore SLO to the initial target. We can monitor the impact of reducingminCapacityto 25 by checking the following dashboards: