Sidecar: Ask if collectors should be restarted after a configuration change

[lennartkoopmann]

I changed the configuration of my osquery instances and wasn't sure if I had to manually restart the collectors next. Worse, all my osqueryd instances failed because the configuration file handle was invalid after the configuration change and I definitely had to restart the processes.

Can we ask the user if the underlying processes should be restarted if a configuration was changed?

[lennartkoopmann]

A possible caveat: We'd have to wait until the new config was applied, or send the restart request together with the config change.

[lennartkoopmann]

User error. The restart is happening automatically already.

Should we still ask?

[mariussturm]

Right, the current behavior is that a configuration change is detected automatically and because of that the Sidecar restarts the process. An option would be to interfere into this procedure and mark a collector with a no-auto-restart flag. The only problem is that the user would need a feedback in the UI if a restart is needed or not. In that way we could roll out configurations and let them validate but the actual restart is up to the user.

[mariussturm]

Thinking this a little bit further, the big boys use-case is in a scenario with lots and lots of collectors. An admin would be afraid to do a configuration change because he doesn't know if all instances will fail or not. The current way of doing it is to create a second collector configuration -> apply it to one or two hosts only, test a configuration change. Afterwards put the changes in the production configuration.

We could support this process by providing a new view, e.g. Collector Workspace where all of this is automated. With one click a configuration and the corresponding collector is copied -> fiddle around -> assign it to one or two hosts -> Promote it to production.

We will try to mock something up to see if that would e feasible way. Probably nothing for 3.0.

[bernd]

I agree, before we think about how we would solve the "ask for restart" we should think about use cases and why someone might want to not restart a collector automatically after a config change. 👍

[kodjo-anipah]

will continue this in Graylog2/collector-sidecar#494