[Ingest Manager] Restart process on output change

[michalpristas]

What does this PR do?

This PR adds an option to specify in a spec file behavior on output change as sometimes new output definition is not reloaded correctly on beats side.

Endpoint reloads output ok, fleet server does not reload output at all but we agreed to fix it on server side.

Why is it important?

#23596

#24538

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Build Cause: Pull request #24907 updated

• Start Time: 2021-04-19T15:27:01.695+0000

• Duration: 70 min 27 sec

• Commit: 412c2bc

Test stats 🧪

Test	Results
Failed	0
Passed	6788
Skipped	16
Total	6804

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	6788
Skipped	16
Total	6804

[elasticmachine]

Pinging @elastic/agent (Team:Agent)

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b restart-on-output upstream/restart-on-output
git merge upstream/master
git push upstream restart-on-output

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b restart-on-output upstream/restart-on-output
git merge upstream/master
git push upstream restart-on-output

[blakerouse]

Change/code looks good, but I still don't like this idea.

Where we going to actually land this approach for 7.13? Have we noticed any side-effects with making this change?

[ph]

@blakerouse You don't like the idea of restarting the process? @urso

[blakerouse]

I would greatly prefer to fix the real issue versus a hack to make it work.

[urso]

I would greatly prefer to fix the real issue versus a hack to make it work.

The output reloading did never work without issues and should not have been used by agent in the first place. This is why I created: #24538

Although I would love to have the reloading issue fixed in Beats, it is no easy fix and we did try a few times in the past. But in order to fix the issue properly we would have to introduce some bigger architectural changes to libbeat, which was decided to postpone. The race condition in the reloading can lead to events not being removed from the memory/disk queue, which will eventually lead to a deadlock (in any Beat).

Although restarting is quite intrusive, we do not expect this to happen every now and so often, right?

[urso]

@michalpristas Have you tried if you can reproduce the issue #23596 before/after the change?

[michalpristas]

@michalpristas Have you tried if you can reproduce the issue #23596 before/after the change?

havent tried but i was not able to repro even before

[blakerouse]

@urso With the new permissions work, any time a new integration is added to a policy that requires a new index or a new permission it will be marked as a new output and so the beats will be restarted.

[urso]

With the new permissions work, any time a new integration is added to a policy that requires a new index or a new permission it will be marked as a new output and so the beats will be restarted.

Yes, I'm aware of that. Getting your environment configured is somewhat bumpy right now. But once you are settled with your setup, reconfigurations of API keys should be an exception.