Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[winlogbeat] Add missing functionality to experimental API #41525

Merged
merged 29 commits into from
Nov 25, 2024
Merged

[winlogbeat] Add missing functionality to experimental API #41525

merged 29 commits into from
Nov 25, 2024

Conversation

marc-gr
Copy link
Contributor

@marc-gr marc-gr commented Nov 5, 2024 •
edited
Loading

Proposed commit message

Makes experimental api on par with the default one:

  • For events that have UserData instead of EventData, populate it also for experimental api
  • Include XML is respected
  • Forwarded events use renderedtext info
  • Language setting is respected
  • Language setting also added to decode xml wineventlog processor
  • Format embedded messages in the experimental api

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

Also added include xml as part of the benchmark matrix, to basically check we perform as the default API when having it enabled:

$ go test -run TestBenchmarkRead -benchmem -benchtime 10s -benchtest -v .
=== RUN   TestBenchmarkRead
=== RUN   TestBenchmarkRead/api=wineventlog
=== RUN   TestBenchmarkRead/api=wineventlog/include_xml=true/batch_size=10
    bench_test.go:119: 3710.24 events/sec        21030 B/event   210302 B/batch  282 allocs/event        2825 allocs/batch
=== RUN   TestBenchmarkRead/api=wineventlog/include_xml=true/batch_size=100
    bench_test.go:119: 4600.02 events/sec        20658 B/event   2065829 B/batch         281 allocs/event        28127 allocs/batch
=== RUN   TestBenchmarkRead/api=wineventlog/include_xml=true/batch_size=500
    bench_test.go:119: 4636.17 events/sec        20301 B/event   10150670 B/batch        281 allocs/event        140554 allocs/batch
=== RUN   TestBenchmarkRead/api=wineventlog/include_xml=true/batch_size=1000
    bench_test.go:119: 4620.39 events/sec        20930 B/event   20930935 B/batch        281 allocs/event        281082 allocs/batch
=== RUN   TestBenchmarkRead/api=wineventlog/include_xml=false/batch_size=10
    bench_test.go:119: 4326.34 events/sec        19713 B/event   197133 B/batch  281 allocs/event        2815 allocs/batch
=== RUN   TestBenchmarkRead/api=wineventlog/include_xml=false/batch_size=100
    bench_test.go:119: 4659.42 events/sec        19391 B/event   1939168 B/batch         280 allocs/event        28026 allocs/batch
=== RUN   TestBenchmarkRead/api=wineventlog/include_xml=false/batch_size=500
    bench_test.go:119: 4647.01 events/sec        19011 B/event   9505945 B/batch         280 allocs/event        140055 allocs/batch
=== RUN   TestBenchmarkRead/api=wineventlog/include_xml=false/batch_size=1000
    bench_test.go:119: 4578.59 events/sec        19644 B/event   19644712 B/batch        280 allocs/event        280085 allocs/batch
=== RUN   TestBenchmarkRead/api=wineventlog-experimental
=== RUN   TestBenchmarkRead/api=wineventlog-experimental/include_xml=true/batch_size=10
    bench_test.go:119: 4191.40 events/sec        21941 B/event   219410 B/batch  281 allocs/event        2817 allocs/batch
=== RUN   TestBenchmarkRead/api=wineventlog-experimental/include_xml=true/batch_size=100
    bench_test.go:119: 4410.18 events/sec        21652 B/event   2165202 B/batch         281 allocs/event        28123 allocs/batch
=== RUN   TestBenchmarkRead/api=wineventlog-experimental/include_xml=true/batch_size=500
    bench_test.go:119: 4509.38 events/sec        21278 B/event   10639211 B/batch        281 allocs/event        140572 allocs/batch
=== RUN   TestBenchmarkRead/api=wineventlog-experimental/include_xml=true/batch_size=1000
    bench_test.go:119: 4480.46 events/sec        21941 B/event   21941332 B/batch        281 allocs/event        281122 allocs/batch
=== RUN   TestBenchmarkRead/api=wineventlog-experimental/include_xml=false/batch_size=10
    bench_test.go:119: 32061.20 events/sec       4324 B/event    43242 B/batch   28 allocs/event         283 allocs/batch
=== RUN   TestBenchmarkRead/api=wineventlog-experimental/include_xml=false/batch_size=100
    bench_test.go:119: 46401.75 events/sec       3917 B/event    391734 B/batch  27 allocs/event         2767 allocs/batch
=== RUN   TestBenchmarkRead/api=wineventlog-experimental/include_xml=false/batch_size=500
    bench_test.go:119: 48948.90 events/sec       3554 B/event    1777464 B/batch         27 allocs/event         13792 allocs/batch
=== RUN   TestBenchmarkRead/api=wineventlog-experimental/include_xml=false/batch_size=1000
    bench_test.go:119: 48948.13 events/sec       4246 B/event    4246701 B/batch         27 allocs/event         27580 allocs/batch
--- PASS: TestBenchmarkRead (262.38s)
    --- PASS: TestBenchmarkRead/api=wineventlog (98.03s)
        --- PASS: TestBenchmarkRead/api=wineventlog/include_xml=true/batch_size=10 (12.52s)
        --- PASS: TestBenchmarkRead/api=wineventlog/include_xml=true/batch_size=100 (14.23s)
        --- PASS: TestBenchmarkRead/api=wineventlog/include_xml=true/batch_size=500 (10.89s)
        --- PASS: TestBenchmarkRead/api=wineventlog/include_xml=true/batch_size=1000 (12.12s)
        --- PASS: TestBenchmarkRead/api=wineventlog/include_xml=false/batch_size=10 (12.10s)
        --- PASS: TestBenchmarkRead/api=wineventlog/include_xml=false/batch_size=100 (13.94s)
        --- PASS: TestBenchmarkRead/api=wineventlog/include_xml=false/batch_size=500 (10.87s)
        --- PASS: TestBenchmarkRead/api=wineventlog/include_xml=false/batch_size=1000 (11.37s)
    --- PASS: TestBenchmarkRead/api=wineventlog-experimental (135.84s)
        --- PASS: TestBenchmarkRead/api=wineventlog-experimental/include_xml=true/batch_size=10 (11.73s)
        --- PASS: TestBenchmarkRead/api=wineventlog-experimental/include_xml=true/batch_size=100 (14.35s)
        --- PASS: TestBenchmarkRead/api=wineventlog-experimental/include_xml=true/batch_size=500 (11.10s)
        --- PASS: TestBenchmarkRead/api=wineventlog-experimental/include_xml=true/batch_size=1000 (12.49s)
        --- PASS: TestBenchmarkRead/api=wineventlog-experimental/include_xml=false/batch_size=10 (24.58s)
        --- PASS: TestBenchmarkRead/api=wineventlog-experimental/include_xml=false/batch_size=100 (17.02s)
        --- PASS: TestBenchmarkRead/api=wineventlog-experimental/include_xml=false/batch_size=500 (22.27s)
        --- PASS: TestBenchmarkRead/api=wineventlog-experimental/include_xml=false/batch_size=1000 (22.29s)
PASS
ok      github.com/elastic/beats/v7/winlogbeat/eventlog 262.417s

@marc-gr marc-gr added enhancement Winlogbeat Team:Security-Windows Platform Windows Platform Team in Security Solution backport-8.x Automated backport to the 8.x branch with mergify labels Nov 5, 2024
@marc-gr marc-gr requested a review from andrewkroh November 5, 2024 14:56
@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Nov 5, 2024
@mergify mergify bot assigned marc-gr Nov 5, 2024
@marc-gr marc-gr marked this pull request as ready for review November 5, 2024 15:38
@marc-gr marc-gr requested a review from a team as a code owner November 5, 2024 15:38
@elasticmachine
Copy link
Collaborator

Pinging @elastic/sec-windows-platform (Team:Security-Windows Platform)

@marc-gr marc-gr changed the title [winlogbeat] Put data under UserData also in experimental api [winlogbeat] Add missing functionality to experimental API Nov 11, 2024
@marc-gr marc-gr mentioned this pull request Sep 20, 2024
Closed
7 tasks
@mergify Mergify
Copy link
Contributor

mergify bot commented Nov 18, 2024

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b feat/experimental-api-userdata upstream/feat/experimental-api-userdata
git merge upstream/main
git push upstream feat/experimental-api-userdata

intxgo
intxgo approved these changes Nov 21, 2024
View reviewed changes
winlogbeat/sys/wineventlog/renderer.go Outdated Show resolved Hide resolved
@marc-gr marc-gr merged commit 4278366 into elastic:main Nov 25, 2024
182 of 183 checks passed
@marc-gr marc-gr deleted the feat/experimental-api-userdata branch November 25, 2024 10:35
mergify bot pushed a commit that referenced this pull request Nov 25, 2024
@marc-gr @mergify
[winlogbeat] Add missing functionality to experimental API (#41525)
7706feb 
* Put data under UserData also in experimental api

* Change docs and changelog

* check evt meta

* Propagate locale config appropiately

* Extract metadata cache

* Add render config

* Simplify render functions

* Add xml rendering to experimental api

* Add benchmarks

* Update docs

* Fix multi os build

* Format embedded messages in the experimental api

* Safer assert

* Test exp api include xml with same test suite

* Check for nil metadata

* Revert "Safer assert"

This reverts commit db5a57d.

* Use single buffer to render xml

(cherry picked from commit 4278366)
@mergify mergify bot mentioned this pull request Nov 25, 2024
Merged
6 tasks
marc-gr added a commit that referenced this pull request Nov 25, 2024
@mergify @marc-gr
[winlogbeat] Add missing functionality to experimental API (#41525) (#…
7e5b5f5 
…41769)

* Put data under UserData also in experimental api

* Change docs and changelog

* check evt meta

* Propagate locale config appropiately

* Extract metadata cache

* Add render config

* Simplify render functions

* Add xml rendering to experimental api

* Add benchmarks

* Update docs

* Fix multi os build

* Format embedded messages in the experimental api

* Safer assert

* Test exp api include xml with same test suite

* Check for nil metadata

* Revert "Safer assert"

This reverts commit db5a57d.

* Use single buffer to render xml

(cherry picked from commit 4278366)

Co-authored-by: Marc Guasch <[email protected]>
Kavindu-Dodan pushed a commit to Kavindu-Dodan/beats that referenced this pull request Nov 27, 2024
@marc-gr @Kavindu-Dodan
[winlogbeat] Add missing functionality to experimental API (elastic#4…
0e4df9b 
…1525)

* Put data under UserData also in experimental api

* Change docs and changelog

* check evt meta

* Propagate locale config appropiately

* Extract metadata cache

* Add render config

* Simplify render functions

* Add xml rendering to experimental api

* Add benchmarks

* Update docs

* Fix multi os build

* Format embedded messages in the experimental api

* Safer assert

* Test exp api include xml with same test suite

* Check for nil metadata

* Revert "Safer assert"

This reverts commit db5a57d.

* Use single buffer to render xml
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@intxgo intxgo intxgo approved these changes

@andrewkroh andrewkroh Awaiting requested review from andrewkroh

Assignees

@marc-gr marc-gr

Labels
backport-8.x Automated backport to the 8.x branch with mergify enhancement Team:Security-Windows Platform Windows Platform Team in Security Solution Winlogbeat
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[winlogbeat] message ids not expanded in individual fields
3 participants
@marc-gr @elasticmachine @intxgo