[winlogbeat] Retry EvtSubscribe from start if fails with strict mode

[marc-gr]

What does this PR do?

Creates a subscription to the winlog channel using EvtSubscribeStrict flag to be able to notice when a bookmark ran invalid. In that scenario we retry the subscription from the beginning of the channel.

Why is it important?

Based on #29793 there is the suspicion that under some scenarios winlogbeat might be ignoring some events when an invalid bookmark is used.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
  - [ ] I have made corresponding changes to the documentation
  - [ ] I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

• Closes [Winlogbeat] Add config option to experiment with EvtSubscribeStrict #29793

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[marc-gr]

@andrewkroh the retry is performed automatically in this first approach. Do you think adding a way to enable or disable this behavior at will would be preferred?

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-03-29T07:37:52.934+0000

• Duration: 74 min 12 sec

Test stats 🧪

Test	Results
Failed	0
Passed	21973
Skipped	1934
Total	23907

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[andrewkroh]

That looks pretty simple so perhaps it does not need to be configurable. Can you create an integration test with windows that exercises this path. One idea I have is to read from a test log we create during the tests, modify the bookmark we have to move the read index forward such that it becomes an invalid bookmark, then try to read using the bookmark, verify it starts at the beginning.

[andrewkroh]

Based on #30201 I think another test case for this should be reading from a channel, deleting the channel, then recreating the channel and read from it again with the previously saved bookmark. I think that's a realistic scenario that is happening to users based on that report.

[nicpenning]

Let me know if I should create a new issue but have you ever noticed that Winlogbeat can't read from the SysMon channel when SysMon is recently installed or when the configuration changes? A restart of the winlogbeat service is needed when making changes to SysMon. It's been many versions since I tested this but I will report back if that's still happening. The only way we could successfully deploy 9k+ winlogbeat agents and SysMon was to install sysmon first then winlogbeat. Then any changes to SysMon would require us to restart winlogbeat so we can gather events as soon as possible.

[marc-gr]

/test

[andrewkroh]

Can you fix this message please.

{"log.level":"debug","@timestamp":"2022-02-09T13:11:48.498Z","log.logger":"winlogbeat","log.origin":{"file.name":"beater/winlogbeat.go","file.line":101},"message":"Initialized EventLog]%!(EXTRA string=WinlogbeatTestPython_79371)","service.name":"winlogbeat","ecs.version":"1.6.0"}

And I wonder if your test is failing because the path name is too long? Trying shortening the test name.

Exiting: rename C:\Users\jenkins\workspace\PR-30155-6-ef9ac65f-57be-49e2-bb4b-e126f9421437\src\github.com\elastic\beats\winlogbeat\build\system-tests\run\test_wineventlog.Test.test_restart_if_bookmarked_event_does_not_exist\data.winlogbeat.yml.new C:\Users\jenkins\workspace\PR-30155-6-ef9ac65f-57be-49e2-bb4b-e126f9421437\src\github.com\elastic\beats\winlogbeat\build\system-tests\run\test_wineventlog.Test.test_restart_if_bookmarked_event_does_not_exist\data.winlogbeat.yml: Access is denied.

[marc-gr]

/test