Scale DCR vertically instead of horizontally

[ioannakok]

Investigate why we are currently using t4g.small, min: 9, max: 36. The goal would be to scale vertically instead of horizontally.

[VDuczekW]

it is a should - we have discussed and this is to visibility - could this wait for CDK migration?
What matrics are we trying to shift

[alinaboghiu]

I think cost is the main metric here and AWS provides dashboards for that on the account home screen:
https://us-east-1.console.aws.amazon.com/cost-management/home?region=eu-west-1#/dashboard

Another metric can be latency and availability:
https://metrics.gutools.co.uk/d/zM0ouzs4z/sli-dashboard-debug-view?orgId=1&var-Period=60&var-AWSAccount=frontend&var-StageFilter=PROD&var-ElkIndexFilter=logstash-frontend-%2A&var-ElasticSearchIndex=logstash-frontend-%2A&var-Service=frontend-PROD-rendering-ELB&var-App=frontend-cfn&var-Stack=frontend&var-Stage=PROD&var-ASGName=frontend-rendering-PROD-AutoscalingGroup-MF5R8QXZY1EX&from=now-12h&to=now

[arelra]

The main issue with scaling vertically is getting DCR (a single threaded node application) to make use of all the available cpus. Enabling cluster mode (via PM2 or a custom implementation) allows this.

A load testing spike analysing latencies with cluster mode enabled for different instance sizes:
#8789

An incident investigation following enabling vertical scaling on our current instance size:
https://docs.google.com/document/d/1XWVUl1P3WoRoZsK6nx6b8FwjpMKXbwf8P1CB-VtEi64/edit

[shesah]

duplicate of #8345