🔍 Investigation Needed: Incident #386206 and Auto-Resolution

[murad-ali-MoJ]

Description

When I was on support on 02/02/24, I consistently received emails from PagerDuty indicating that I had an open incident assigned to me:

INCIDENT: #386206
Cluster: rds-eks-production-control-panel-psg-db-encrypted-low-freeable-memory

Issue Details

However, upon checking the incident details, it appears to be resolved. I suspect there might have been a memory issue in one of the clusters, and the system may have automatically resolved it.

Steps to Reproduce

N/A (incident was automatically resolved)

Expected Behavior

Provide insights or investigation on the potential memory issue in the mentioned cluster.

Additional Information

• Link to PagerDuty Incident Details
• More logs or error messages

[jacobwoffenden]

Summary

Control Panel's RDS instance is db.t3.micro (2vCPU, 1GB RAM) https://github.com/ministryofjustice/data-platform/blob/4b9253cd26e7f494b3374abc1064ee543941dc6e/terraform/aws/analytical-platform-production/cluster/terraform.tfvars#L54

The alarm is configured https://github.com/ministryofjustice/data-platform/blob/4b9253cd26e7f494b3374abc1064ee543941dc6e/terraform/aws/analytical-platform-production/cluster/cloudwatch-alarms.tf#L223-L240 to alert when freeable memory drops below 128MB https://github.com/ministryofjustice/data-platform/blob/4b9253cd26e7f494b3374abc1064ee543941dc6e/terraform/aws/analytical-platform-production/cluster/terraform.tfvars#L102

128MB is a sufficient threshold to alert on for a database with 1GB RAM

Suggested Action

Migrate to db.t4g.small (2 vCPU, 2GB RAM) or db.t4g.medium (2 vCPU, 4GB RAM)

[BrianEllwood]

looking at the logs this issue looks to have stopped on the 5/2/24

when the freeable memory jumped from an average of around 127M to 212M.

I have not been able to find a reason for this in any logs still available.

[BrianEllwood]

Looking at the documentation for a RDS PostgreSQL instance there does not appear to be any memory structures that could be tuned to help with this alert when it occurs.

If we did want to fix this (not currently occurring) issue we would have to increase the instance size as suggested.

[BrianEllwood]

Looking at the incidents raised in pager duty i can see that this issue was first alerted 17/1/24

[BrianEllwood]

As there is currently no issue I would advise to take no action and investigate if it reoccurs in future. I will move the ticket into review and if the team agress I will close the ticket.

If not i will increase the instance size in the dev cluster as a test before implementing into prod.

[BrianEllwood]

Closing this ticket, new issue raised to look at log retention here