Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[7.17](backport #30942) winlogbeat/sys/wineventlog: extend testing and fix bugs #31096

Merged
merged 4 commits into from
May 3, 2022

Conversation

mergify[bot]
Copy link
Contributor

@mergify mergify bot commented Mar 31, 2022

This is an automatic backport of pull request #30942 done by Mergify.
Cherry-pick of 34bdc3d has failed:

On branch mergify/bp/7.17/pr-30942
Your branch is up to date with 'origin/7.17'.

You are currently cherry-picking commit 34bdc3d468.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   CHANGELOG.next.asciidoc
	modified:   winlogbeat/eventlog/wineventlog_test.go
	modified:   winlogbeat/sys/winevent/event.go
	modified:   winlogbeat/sys/winevent/event_test.go
	modified:   winlogbeat/sys/wineventlog/format_message.go
	modified:   winlogbeat/sys/wineventlog/query_test.go
	modified:   winlogbeat/sys/wineventlog/renderer.go
	modified:   winlogbeat/sys/wineventlog/renderer_test.go
	new file:   winlogbeat/sys/wineventlog/testdata/application-windows-error-reporting.xml
	new file:   winlogbeat/sys/wineventlog/testdata/ec1.evtx
	new file:   winlogbeat/sys/wineventlog/testdata/ec1.xml
	new file:   winlogbeat/sys/wineventlog/testdata/ec2.evtx
	new file:   winlogbeat/sys/wineventlog/testdata/ec2.xml
	new file:   winlogbeat/sys/wineventlog/testdata/ec3.evtx
	new file:   winlogbeat/sys/wineventlog/testdata/ec3.xml
	new file:   winlogbeat/sys/wineventlog/testdata/ec3and4.evtx
	new file:   winlogbeat/sys/wineventlog/testdata/ec3and4.xml
	new file:   winlogbeat/sys/wineventlog/testdata/ec4.evtx
	new file:   winlogbeat/sys/wineventlog/testdata/ec4.xml
	new file:   winlogbeat/sys/wineventlog/testdata/experimental.evtx
	new file:   winlogbeat/sys/wineventlog/testdata/experimental.xml
	new file:   winlogbeat/sys/wineventlog/testdata/original.evtx
	new file:   winlogbeat/sys/wineventlog/testdata/original.xml
	new file:   winlogbeat/sys/wineventlog/testdata/sysmon-9.01.xml
	modified:   winlogbeat/sys/wineventlog/wineventlog_windows_test.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   winlogbeat/Jenkinsfile.yml
	both modified:   winlogbeat/beater/winlogbeat.go
	both modified:   winlogbeat/eventlog/wineventlog.go
	both modified:   winlogbeat/eventlog/wineventlog_experimental.go
	both modified:   winlogbeat/sys/wineventlog/wineventlog_windows.go
	both modified:   x-pack/winlogbeat/Jenkinsfile.yml

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/github/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally


Mergify commands and options

More conditions and actions can be found in the documentation.

You can also trigger Mergify actions by commenting on this pull request:

  • @Mergifyio refresh will re-evaluate the rules
  • @Mergifyio rebase will rebase this PR on its base branch
  • @Mergifyio update will merge the base branch into this PR
  • @Mergifyio backport <destination> will backport this PR on <destination> branch

Additionally, on Mergify dashboard you can:

  • look at your merge queues
  • generate the Mergify configuration with the config editor.

Finally, you can contact us on https://mergify.com

This fixes failures in event handling on Windows 2022[1] where parts of
events available from the Windows API are not reflected in the events
recovered by winlogbeat[2].

There is unfortunately quite a lot of movement in this change due to the
need to satisfy linter requirements. Beyond those changes, the substantive
changes here are:

1. Addition of new testing infrastructure to allow addition of evtx files
   and comparison with there expected XML renderings, and adding some test
   cases (shown below).
2. Fixing a buffer length parameter in the call to _EvtFormatMessage in
   evtFormatMessage that was the result of a lack of clarity in the API
   documentation for that syscall.
3. Fixing a var shadowing decl of the publisher handle EvtHandle in
   FormatEventString.
4. Providing a call back for the legacy (non-experimental) API through
   wineventlog.Message to allow it to obtain the event message in the case
   that the RenderingInfo element is not available via the Windows API.
5. Ensure that keyword, opcode and level are obtained by the non-experimental
   API by calling winevent.EnrichRawValuesWithNames in buildRecordFromXML.
   This change also required making winevent.Event.OpcodeRaw a pointer to
   allow an absent System>Opcode element to be distinquished from the zero,
   but present element.

The change also enables testing on Windows 2022.

[1]#30621
[2]#30622 (comment)

New events in testing as rendered by the Event Viewer:

ec1: eventcreate /id 1000 /t error /l application /d "My custom error event for the application log"

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
          <Provider Name="EventCreate" />
          <EventID Qualifiers="0">1000</EventID>
          <Version>0</Version>
          <Level>2</Level>
          <Task>0</Task>
          <Opcode>0</Opcode>
          <Keywords>0x80000000000000</Keywords>
          <TimeCreated SystemTime="2022-03-22T02:03:11.3106672Z" />
          <EventRecordID>316</EventRecordID>
          <Correlation />
          <Execution ProcessID="0" ThreadID="0" />
          <Channel>Application</Channel>
          <Computer>vagrant</Computer>
          <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" />
      </System>
    - <EventData>
          <Data>My custom error event for the application log</Data>
      </EventData>
  </Event>

ec2: eventcreate /id 999 /t error /l application /so WinWord /d "Winword event 999 happened due to low diskspace"

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
          <Provider Name="WinWord" />
          <EventID Qualifiers="0">999</EventID>
          <Version>0</Version>
          <Level>2</Level>
          <Task>0</Task>
          <Opcode>0</Opcode>
          <Keywords>0x80000000000000</Keywords>
          <TimeCreated SystemTime="2022-03-22T02:03:11.5132246Z" />
          <EventRecordID>317</EventRecordID>
          <Correlation />
          <Execution ProcessID="0" ThreadID="0" />
          <Channel>Application</Channel>
          <Computer>vagrant</Computer>
          <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" />
      </System>
    - <EventData>
          <Data>Winword event 999 happened due to low diskspace</Data>
      </EventData>
  </Event>

ec3: eventcreate /id 5 /t error /l system /d "Catastrophe!"

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
          <Provider Name="system" />
          <EventID Qualifiers="0">5</EventID>
          <Version>0</Version>
          <Level>2</Level>
          <Task>0</Task>
          <Opcode>0</Opcode>
          <Keywords>0x80000000000000</Keywords>
          <TimeCreated SystemTime="2022-03-22T02:03:11.5455572Z" />
          <EventRecordID>1413</EventRecordID>
          <Correlation />
          <Execution ProcessID="0" ThreadID="0" />
          <Channel>System</Channel>
          <Computer>vagrant</Computer>
          <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" />
      </System>
    - <EventData>
          <Data>Catastrophe!</Data>
      </EventData>
  </Event>

ec4: eventcreate /id 5 /t error /l system /so Backup /d "Backup failure"

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
          <Provider Name="Backup" />
          <EventID Qualifiers="0">5</EventID>
          <Version>0</Version>
          <Level>2</Level>
          <Task>0</Task>
          <Opcode>0</Opcode>
          <Keywords>0x80000000000000</Keywords>
          <TimeCreated SystemTime="2022-03-22T02:03:11.8616638Z" />
          <EventRecordID>1414</EventRecordID>
          <Correlation />
          <Execution ProcessID="0" ThreadID="0" />
          <Channel>System</Channel>
          <Computer>vagrant</Computer>
          <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" />
      </System>
    - <EventData>
          <Data>Backup failure</Data>
      </EventData>
  </Event>

Co-authored-by: Andrew Kroh <[email protected]>
(cherry picked from commit 34bdc3d)

# Conflicts:
#	winlogbeat/Jenkinsfile.yml
#	winlogbeat/beater/winlogbeat.go
#	winlogbeat/eventlog/wineventlog.go
#	winlogbeat/eventlog/wineventlog_experimental.go
#	winlogbeat/sys/wineventlog/wineventlog_windows.go
#	x-pack/winlogbeat/Jenkinsfile.yml
@mergify mergify bot requested a review from a team as a code owner March 31, 2022 20:35
@mergify mergify bot added backport conflicts There is a conflict in the backported pull request labels Mar 31, 2022
@mergify mergify bot assigned efd6 Mar 31, 2022
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Mar 31, 2022
@elasticmachine
Copy link
Collaborator

elasticmachine commented Mar 31, 2022

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2022-05-03T01:20:30.271+0000

  • Duration: 90 min 23 sec

Test stats 🧪

Test Results
Failed 0
Passed 20219
Skipped 1545
Total 21764

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

@efd6
Copy link
Contributor

efd6 commented Mar 31, 2022

Blocked on #31042

@mergify
Copy link
Contributor Author

mergify bot commented Apr 4, 2022

This pull request has not been merged yet. Could you please review and merge it @efd6? 🙏

2 similar comments
@mergify
Copy link
Contributor Author

mergify bot commented Apr 11, 2022

This pull request has not been merged yet. Could you please review and merge it @efd6? 🙏

@mergify
Copy link
Contributor Author

mergify bot commented Apr 18, 2022

This pull request has not been merged yet. Could you please review and merge it @efd6? 🙏

@mergify
Copy link
Contributor Author

mergify bot commented Apr 20, 2022

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b mergify/bp/7.17/pr-30942 upstream/mergify/bp/7.17/pr-30942
git merge upstream/7.17
git push upstream mergify/bp/7.17/pr-30942

@mergify mergify bot requested a review from a team as a code owner April 21, 2022 02:44
@mergify mergify bot requested review from belimawr and kvch and removed request for a team April 21, 2022 02:44
@efd6 efd6 force-pushed the mergify/bp/7.17/pr-30942 branch from 91aad25 to 534def1 Compare April 21, 2022 04:00
@mergify
Copy link
Contributor Author

mergify bot commented Apr 25, 2022

This pull request has not been merged yet. Could you please review and merge it @efd6? 🙏

@efd6
Copy link
Contributor

efd6 commented May 2, 2022

Depends on parts of #31464 which won't be back ported to this change (they will be cherry picked to here when that PR is approved).

  1. The change in winlogbeat/eventlog/wineventlog.go adding meta data to the winevent.EnrichRawValuesWithNames call.
  2. The change in x-pack/winlogbeat/module/powershell/test/powershell_windows_test.go conditionally ignoring the winlog.opcode field on Windows 2022 and dependent change in x-pack/winlogbeat/module/testing_windows.go to block golden file generation on that version.

@mergify
Copy link
Contributor Author

mergify bot commented May 2, 2022

This pull request has not been merged yet. Could you please review and merge it @efd6? 🙏

@efd6
Copy link
Contributor

efd6 commented May 3, 2022

@andrewkroh Please take a quick look at the last commit here; it's the relevant parts of #31464 for 7.17 to get 2022 to pass. It's large but the same as happened there less the recovery of testing.

@efd6 efd6 merged commit 953f799 into 7.17 May 3, 2022
@mergify mergify bot deleted the mergify/bp/7.17/pr-30942 branch May 3, 2022 21:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport conflicts There is a conflict in the backported pull request needs_team Indicates that the issue/PR needs a Team:* label
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants