Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Reload (instead of restart) Prometheus service if configs change and service is already running #80

Open
mdschmitt opened this issue Jul 1, 2022 · 1 comment · May be fixed by #83
Labels
enhancement New feature or request

Comments

@mdschmitt
Copy link
Contributor

Is your feature request related to a problem?

There's actually no need to restart the prometheus service if you're just updating the configs. A simple reload will do the trick. https://prometheus.io/docs/introduction/faq/#can-i-reload-prometheuss-configuration

Describe the solution you'd like

IF

  1. The prometheus service and configurations files already exist (i.e. are just being updated/changed, as opposed to being freshly created)

AND

  1. The prometheus service is already running

THEN

Reload the prometheus service instead of "hard" restarting it.

Describe alternatives you've considered

I guess maybe enabling --enable-feature=memory-snapshot-on-shutdown https://prometheus.io/docs/prometheus/latest/feature_flags/#memory-snapshot-on-shutdown would help minimize restart time, but it's technically an experimental feature which makes me think I shouldn't run it in prod..

Additional context

If your prometheus installation is large-scale (i.e. scrapes a LOT of data), restarting it can take minutes while it replays the WAL logs. Reloading involves no downtime.

@mdschmitt mdschmitt added the enhancement New feature or request label Jul 1, 2022
@mdschmitt mdschmitt changed the title [FEATURE] _Reload_ (instead of restart) Prometheus service if configs change and service is already running [FEATURE] Reload (instead of restart) Prometheus service if configs change and service is already running Jul 1, 2022
@mdschmitt mdschmitt linked a pull request Aug 26, 2022 that will close this issue
19 tasks
@mdschmitt
Copy link
Contributor Author

ugh, it seems that if the rules are changed, a reload is not enough, the service must be restarted. Back to the memory snapshot feature, I guess, unless I create a whole separate service.running resource for restarting instead of reloading.

https://saltstackcommunity.slack.com/archives/C7K04SEJC/p1662430673735549

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant