Pin prometheus chart version to an older version #720
Conversation
|
I'm currently trying to deploy it locally with: to see if it works. Update: Hmm, not sure it's normal for it to take this much time. |
|
Running So, I guess I need to manually delete the old pod? |
|
@GeorgianaElena yep, you should manually delete it. Prometheus should have config set like this to prevent this from happening. When the strategy is set to |
|
Thanks @yuvipanda! The deployment went ok after deleting the pod 🎉 However, it doesn't look like anything's changed in https://grafana.pilot.2i2c.cloud. Is there anything else I should do, or the dashboards should just have magically appeared if this |
|
@GeorgianaElena I think github.com/jupyterhub/jupyterhub-grafana need to be deployed there to see if it works. Can you give that a shot too? |
I just did and I think they work 🎉 |
|
Yay awesome! We need to also automate deployment of rhe gradana dashboard too |
|
yesssss dashboards!!! thanks @GeorgianaElena @yuvipanda I assume you mean that automation should be tackled in the future, not as part of this PR? |
|
@choldgraf yep! |
I quickly searched about grafana and prometheus open issues and there are several of them. |
During deployment of 2i2c-org#720, sometimes CI would fail because the prometheus pod would be stuck in 'ContainerCreating', as the old pod was holding on to the persistent disk the new pod needs to start. This was temporarily fixed by deleting the prometheus pod, but this tells kubernetes to delete the old pod properly first before starting the new one. Ref 2i2c-org#720 (comment)
During deployment of 2i2c-org#720, sometimes CI would fail because the prometheus pod would be stuck in 'ContainerCreating', as the old pod was holding on to the persistent disk the new pod needs to start. This was temporarily fixed by deleting the prometheus pod, but this tells kubernetes to delete the old pod properly first before starting the new one. Ref 2i2c-org#720 (comment)
|
@damianavila I don't think that I follow. Do you mean have a high-level issue about grafana/prometheus specifically? @yuvipanda what are the things that we need to do in order to automatically deploy Grafana for our hubs? Is that something we should track in a new issue? |
|
@choldgraf i opened #739 to track that |
I was thinking more about a sort of "topic" label? The Managed JupyterHubs Infrastructure column is pretty big so some filtering by label would help to visibilize related stuff... |
|
@damianavila ahhh - yes, I totally agree. That column has way more stuff in general than all of the other columns and I also find it hard to parse 😅. Another option is to create a label like 🏷️ reporting Would that make sense? |
|
|
|
I was proposing that the topic we use to describe "infra to get info about our infra" is "reporting" |
|
Thanks for the clarification... |
|
fair enough - what about |
|
|
|
I think 'monitoring' is a more common industry standard term for this. https://sre.google/sre-book/monitoring-distributed-systems/ is a nice read |
|
Yep, 💯 to |
|
ah I knew there was a better word haha, thanks - will go with that |
Fixes #616