Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Review DCR Scaling Policy #9311

Closed
1 task done
Tracked by #8345
ioannakok opened this issue Oct 25, 2023 · 5 comments
Closed
1 task done
Tracked by #8345

Review DCR Scaling Policy #9311

ioannakok opened this issue Oct 25, 2023 · 5 comments
Assignees
Milestone

Comments

@ioannakok
Copy link
Contributor

ioannakok commented Oct 25, 2023

Ticket created from #8345 (comment)

This ticket is to:

  1. Experiment with alternative solutions on the DCR scaling policy
  2. Review the metric upon which we want to base our scaling on.

DCR Scaling Policy

DCR is currently using Simple Scaling Policy1. AWS have essentially deprecated this style of scaling and are promoting their alternative solutions as a best practice:

image

Do we want to consider adopting one of the other alternatives?

Some examples in other Guardian repos:

DCR Scaling Policy Metric

We're currently scaling based on latency:

const latencyScalingAlarmConfig: DCRAlarmConfig = {
threshold: 0.2,
period: 60,
evaluationPeriod: 1,
comparisonOperator: 'GreaterThanOrEqualToThreshold',
};

and scaling up by doubling our capacity every 10 minutes:

const scaleUpPolicy = new CfnScalingPolicy(this, 'ScaleUpPolicy', {
adjustmentType: AdjustmentType.PERCENT_CHANGE_IN_CAPACITY,
autoScalingGroupName: asg.autoScalingGroupName,
cooldown: '600',
scalingAdjustment: 100,
});

whilst scaling down by removing an instance once every 2 minutes:

const scaleDownPolicy = new CfnScalingPolicy(this, 'ScaleDownPolicy', {
adjustmentType: AdjustmentType.CHANGE_IN_CAPACITY,
autoScalingGroupName: asg.autoScalingGroupName,
cooldown: '120',
scalingAdjustment: -1,
});

Do we want to consider other scaling strategies? Apps-rendering, for example, via guardian/cdk, scales based on a target CPU utilisation:

asg.scaleOnCpuUtilization('CpuScalingPolicy', {
targetUtilizationPercent: scalingTargetCpuUtilisation,
});

Tasks

Preview Give feedback
  1. 3 of 3
    JamieB-gu

Footnotes

  1. Simple scaling is the default, so if PolicyType is unspecified, SimpleScaling is used.

@ioannakok
Copy link
Contributor Author

@ioannakok ioannakok moved this from In progress to Planned in WebX Team Nov 7, 2023
@ioannakok ioannakok moved this from Planned to Backlog in WebX Team Nov 21, 2023
@alinaboghiu alinaboghiu moved this from Backlog to Planned in WebX Team Dec 13, 2023
@alinaboghiu alinaboghiu moved this from Planned to In progress in WebX Team Dec 13, 2023
@cemms1 cemms1 moved this from In Progress to Review in WebX Team Jan 15, 2024
@cemms1
Copy link
Contributor

cemms1 commented Jan 15, 2024

Implemented a step scaling policy in #10127

@cemms1 cemms1 moved this from Review to Backlog in WebX Team Jan 16, 2024
@cemms1
Copy link
Contributor

cemms1 commented Jan 16, 2024

Moving to backlog after merging #10127 since there are further opportunities to improve the scaling logic and these can be addressed in future health weeks

@arelra arelra moved this from Backlog to In Progress in WebX Team Jun 26, 2024
@arelra arelra self-assigned this Jun 26, 2024
@arelra
Copy link
Member

arelra commented Jun 26, 2024

Re-opening here:
#11715

@shesah shesah moved this from In Progress to This Sprint in WebX Team Jul 2, 2024
@shesah shesah moved this from This Sprint to In Progress in WebX Team Jul 8, 2024
@mxdvl mxdvl changed the title Review DCR Scaling Policy Review DCR Scaling Policy & Provisioning Jul 8, 2024
@mxdvl mxdvl changed the title Review DCR Scaling Policy & Provisioning Review DCR Scaling Policy Jul 8, 2024
@arelra
Copy link
Member

arelra commented Jul 22, 2024

Completed by:
#11837
#11928

@arelra arelra closed this as completed Jul 22, 2024
@github-project-automation github-project-automation bot moved this from In Progress to Done in WebX Team Jul 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Development

No branches or pull requests

5 participants