Review DCR's ASG Configuration

[JamieB-gu]

Now that we're serving fronts traffic it might be a good time to review, to establish whether we're:

1. Using the best instance types for our use-case
2. Scaled to an appropriate count, with adequate min and max counts that allow for RiffRaff deploys
3. Applying the best scaling policy for our needs

Suggestions derived from conversations with @alinaboghiu and @jacobwinch .

Tasks

Preview Give feedback

1. Review ASG Sizes #9322
   +0
   

   
2. Review DCR Scaling Policy #9311
   1 of 1
   +0
   
3. Review DCR Instance Types #9321
   +0
   

   
4. Migrate from using ELB to ALB Load Balancer Type for DCR #9310
   +0
   
5. SPIKE: Custom cluster mode using Node API #9963
   +0
   

   
6. Once Spin up multiple AWS stacks to stop DCR being a single source of failure in our request path #8351 is complete - revisit each new stack ASG size and instance types

Instance Types

We're currently using t4g.small in production:

dotcom-rendering/dotcom-rendering/scripts/deploy/riff-raff.yaml

Lines 16 to 19 in e234547

	CODE:
	InstanceType: t4g.micro
	PROD:
	InstanceType: t4g.small

Do we want to consider other instance types? Frontend uses a different type, as mentioned in #7440.

ASG Size

We're currently using a minimum of 15 and a maximum of 60 instances in production:

dotcom-rendering/dotcom-rendering/cloudformation.yml

Lines 72 to 78 in e234547

	StageMap:
	PROD:
	MinCapacity: 15
	MaxCapacity: 60
	CODE:
	MinCapacity: 1
	MaxCapacity: 4

Do these limits allow enough headroom for our additional traffic, scaling requirements, and RiffRaff deploys?

Scaling Policy

We're currently scaling based on latency:

dotcom-rendering/dotcom-rendering/cloudformation.yml

Lines 312 to 331 in e234547

	LatencyScalingAlarm:
	Condition: HasLatencyScalingAlarm
	Properties:
	AlarmDescription: !Sub |
	Scale-Up if latency is greater than 0.2 seconds over 1 period(s) of 60 seconds
	Dimensions:
	- Name: LoadBalancerName
	Value: !Ref InternalLoadBalancer
	EvaluationPeriods: '1'
	MetricName: Latency
	Namespace: AWS/ELB
	Period: '60'
	Statistic: Average
	Threshold: '0.2'
	ComparisonOperator: GreaterThanOrEqualToThreshold
	OKActions:
	- !Ref ScaleDownPolicy
	AlarmActions:
	- !Ref ScaleUpPolicy
	Type: AWS::CloudWatch::Alarm

and scaling up by doubling our capacity every 10 minutes:

dotcom-rendering/dotcom-rendering/cloudformation.yml

Lines 304 to 310 in e234547

	ScaleUpPolicy:
	Type: AWS::AutoScaling::ScalingPolicy
	Properties:
	AdjustmentType: PercentChangeInCapacity
	AutoScalingGroupName: !Ref AutoscalingGroup
	Cooldown: '600'
	ScalingAdjustment: '100'

whilst scaling down by removing an instance once every 2 minutes:

dotcom-rendering/dotcom-rendering/cloudformation.yml

Lines 296 to 302 in e234547

	ScaleDownPolicy:
	Type: AWS::AutoScaling::ScalingPolicy
	Properties:
	AdjustmentType: ChangeInCapacity
	AutoScalingGroupName: !Ref AutoscalingGroup
	Cooldown: '120'
	ScalingAdjustment: '-1'

Do we want to consider other scaling strategies? Apps-rendering, for example, via guardian/cdk, scales based on a target CPU utilisation:

dotcom-rendering/apps-rendering/cdk/lib/mobile-apps-rendering.ts

Lines 104 to 106 in e234547

	asg.scaleOnCpuUtilization('CpuScalingPolicy', {
	targetUtilizationPercent: scalingTargetCpuUtilisation,
	});

@akash1810 recommends to have a completed test before Christmas holidays and base on that we decide what to do.

[ParisaTork]

Instance Types
T4G instances are suitable for workloads that don't use the full CPU often or consistently. Given the recent spikes in traffic (I don’t know what DCR’s CPU utilisation was at the time of the incidents, but Facia’s reached 100% (guardian/frontend#26336 (comment))), would it be worth having the C7G instances?

ASG Size
Do our instances scale down quickly when the cache is populated to reduce costs?

Scaling Policy
Do we have an SLA that explicitly sets out expected performance metrics? This would help us determine whether our scaling strategy should be based on latency, CPU utilisation, Network I/O etc. Also is it worth having some scheduled scaling between 12 and 1, when our traffic tends to peak: https://docs.google.com/document/d/1heaaDCNJ45uaZRsIUgJxdOc9vz3z5eIzf4sIMGJ4X44/edit?usp=sharing or is it more cost-effective and performant to have dynamic scaling all the time?

[jamesgorrie]

Something I think that is also worth adding to this discussion is how we might have scaling work in lockstep with frontend.

Out current setup of code, not-colocated, makes it seem as though there is no coupling between this AWS configuration, and frontend's AWS configuration.

I'd suggest there is strong coupling. If we scale up the frontend/facia app, we will send more traffic to dcr. We should account for this as failing at the dcr level will cause just as many issue, especially given its current "I serve HTML for all the things" setup.

We should probably consider that we are planning on being able to scale up separately across endpoints in dcr soon too to avoid that single point of failure.

[jacobwinch]

Thanks for writing this up @JamieB-gu!

3. Applying the best scaling policy for our needs

IIUC this app (and other Dotcom apps) are currently using simple scaling¹. AWS have essentially deprecated this style of scaling and are promoting their alternative solutions as a best practice:

I would recommend experimenting with these alternative strategies (regardless of which metric you decide to base your scaling on) if you have some time.

Footnotes

1. It's a bit tricky to see this, but simple scaling is the default, so if PolicyType is unspecified, you're using SimpleScaling. ↩

[AshCorr]

Worth noting that even though we're paying for a t4g.small instances with 2 vCPUs in PROD I'm not convinced we actually use the 2nd CPU for DCR. Due to Nodes single threaded nature it will only use one of the CPUs, usually something like PM2 would create multiple node instances for each CPU to fix that but I don't think we've configured it to do that in our case.

[arelra]

Worth noting that even though we're paying for a t4g.small instances with 2 vCPUs in PROD I'm not convinced we actually use the 2nd CPU for DCR. Due to Nodes single threaded nature it will only use one of the CPUs, usually something like PM2 would create multiple node instances for each CPU to fix that but I don't think we've configured it to do that in our case.

@AshCorr Snap. I'm looking at configuring clustering

[jamesgorrie]

We should probably consider that we are planning on being able to scale up separately across endpoints in dcr soon too to avoid that single point of failure.

Talking to myself - there is an issue that speaks to this and might be considered in conjunction with this as we can think of scaling as domain and traffic specific to those parts of the site.

[rhiannareechaye]

hey @jamesgorrie I've moved this into the triage column - what do you think the next step is for this piece of work. Do you think it should be prioritised as a 'high impact' ?

[rhiannareechaye]

Moving to the backlog - please note this isn't in our list for planning for the near future, please shout if you disagree though (and feel free to add to the list to be discussed in planning: https://docs.google.com/document/d/1-ls95KamOB-lvwKzTUfqpd3gSwcvgszC7fsMIOhXacM/edit)

[rhiannareechaye]

We'll be prioritising PM2/Vulnerabilities at the moment, and working out a way to surface this on a backlog (we'll try and trial a new approach to health for the new Q)

[ioannakok]

Progress so far: https://docs.google.com/document/d/1FjDANYnddGrB3JnXdMjoHnie2kB7fL3MgJachCVZHVg/edit