Understand and document the steps to safely expand prometheus's disk size, and do so for Pangeo

[sgibson91]

Context

We sometimes need to increase the size of the prometheus disk to accommodate the amount of data we are generating. We should document the steps of how to do this safely so any engineer can perform the task without worrying about running into errors such as below and needing to debug them.

Error: UPGRADE FAILED: cannot patch "support-prometheus-server" with kind PersistentVolumeClaim: persistentvolumeclaims "support-prometheus-server" is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize

When we have these steps, we should use them to merge PR #4808

Task list

Tasks

Preview Give feedback

1. Figure out the steps needed to safely expand prometheus's disk size
2. Document the steps
3. Use the steps to merge Increase size of pangeo's prometheus disk #4808

Definition of Done

• Increase size of pangeo's prometheus disk #4808 is merged and running as expected
• There is documentation for expanding prometheus' disk size that any engineer can follow

Pre-defined Definition of Done

• All the tasks have been completed
• It has been tested to work and to not break any existing functionality
• Documentation was added so other engineers can implement this in the future if needed

[haroldcampbell]

Thi sis dependent on #4810

[sgibson91]

The reason that the steps for expanding the persistentVolume size were not required for pangeo-hubs, is because that cluster is GCP, and all the storage classes there have autoexpansion set to true by default.