Collect data and assess node auto-provisioning on Pangeo cluster

[sgibson91]

Description

Following discussion in #666 we deployed node auto-provisioning on the Pangeo cluster in #670 (since reverted in #678). This issue is to track the performance, cost-benefits, etc of using node auto-provisioning in this cluster and decide whether this is a feature we want to support long-term as an alternative to configuring node pools manually.

This is a time-boxed issue and we should aim to be making a decision within a 2-3 month time frame.

Some more context: #666 (comment)

Value / benefit

✅ Node auto-provisioning would reduce 2i2c Engineer's manual interaction with the cluster
❌ It may not actually be that cost-effective

We don't know the answers to these q's until we gather data and analyse 🙂

Implementation details

No response

Tasks to complete

• Debug node autoprovisioning: did not match Pod's node affinity #677
• Decide what metrics we need to evaluate and what "success" or "failure" of those metrics looks like
• Decide what data will help towards evaluating those metrics
• Decide how that data should be collected
• Implement collection of data
• Analyse data and evaluate metrics
• Decide to support node auto-provisioning long-term or not
• Decide whether to write up a blog post describing this process, the data we collected, and our conclusions

Updates

• Note: This is blocked by Debug node autoprovisioning: did not match Pod's node affinity #677

[choldgraf]

Updated to add the possibility of a little blog post at the end, after chatting about it with @sgibson91 :-)