Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Logs intake fixes #9667

Merged
merged 4 commits into from
Nov 24, 2022
Merged

Logs intake fixes #9667

merged 4 commits into from
Nov 24, 2022

Conversation

axw
Copy link
Member

@axw axw commented Nov 24, 2022

Motivation/summary

Fix some a few small bugs in the logs intake:

  • accept event.dataset, not dataset
  • don't set error.grouping_key for application logs just because they have error.* fields
  • add missing field definitions to app_logs

Checklist

- [ ] Update CHANGELOG.asciidoc

  • Update package changelog.yml (only if changes to apmpackage have been made)
    - [ ] Documentation has been updated

How to test these changes

  1. Follow steps in Add ECS log files to the intake log endpoint #9349
  2. Check that none of the logs have error.grouping_key set
  3. Check that all fields are explicitly mapped; there should be no unknown fields when looking in Discover

Related issues

Closes #9666

axw added 2 commits November 24, 2022 14:44
@axw
modeldecoder: accept "event.dataset"
bde191b 
The intake accepts the ECS field, "event.dataset",
and copies it to the field of the same name in
resulting documents.
@axw
Only set error.grouping_key for error events
13875ce 
Check the processor.* value, don't just take the
presence of error.* fields to mean the event is
an error.
@axw axw added test-plan v8.6.0 backport-8.6 Automated backport with mergify labels Nov 24, 2022
@axw axw changed the title Logs fixes Logs intake fixes Nov 24, 2022
@axw axw marked this pull request as ready for review November 24, 2022 07:01
@axw axw requested a review from a team November 24, 2022 07:01
@apmmachine
Copy link
Contributor

apmmachine commented Nov 24, 2022 •
edited
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2022-11-24T09:00:07.244+0000

  • Duration: 26 min 25 sec

Test stats 🧪

Test Results
Failed 0
Passed 153
Skipped 0
Total 153

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate and publish the docker images.

  • /test windows : Build & tests on Windows.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

lahsivjar
lahsivjar approved these changes Nov 24, 2022
View reviewed changes
Copy link
Contributor

@lahsivjar lahsivjar left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @axw!

@axw axw enabled auto-merge (squash) November 24, 2022 08:59
@apmmachine
Copy link
Contributor

📚 Go benchmark report

Diff with the main branch

name                                                                                              old time/op    new time/op    delta
pkg:github.com/elastic/apm-server/internal/agentcfg goos:linux goarch:amd64
FetchAndAdd/FetchFromCache-12                                                                       41.2ns ± 0%    46.1ns ± 0%   +11.81%  (p=0.016 n=4+5)
FetchAndAdd/FetchAndAddToCache-12                                                                   97.1ns ± 8%   105.4ns ± 0%    +8.51%  (p=0.016 n=5+4)
pkg:github.com/elastic/apm-server/internal/beater/request goos:linux goarch:amd64
ContextResetContentEncoding/empty-12                                                                 123ns ± 2%     136ns ± 0%   +10.51%  (p=0.008 n=5+5)
ContextResetContentEncoding/uncompressed-12                                                          148ns ± 2%     160ns ± 1%    +7.71%  (p=0.008 n=5+5)
pkg:github.com/elastic/apm-server/internal/model/modelindexer goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/processor/stream goos:linux goarch:amd64
BackendProcessorParallel/BenchmarkBackendProcessorParallel4/otel-bridge.ndjson-12                   7.00µs ± 8%    7.71µs ± 5%   +10.05%  (p=0.032 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel4/spans.ndjson-12                         19.4µs ± 3%    21.5µs ± 9%   +11.09%  (p=0.032 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/events.ndjson-12                        15.9µs ± 5%    15.3µs ± 1%    -3.67%  (p=0.032 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/metadata-null-values.ndjson-12          1.18µs ± 1%    1.15µs ± 1%    -2.40%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/minimal-service.ndjson-12               1.41µs ± 4%    1.35µs ± 1%    -4.62%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/optional-timestamps.ndjson-12           1.97µs ± 4%    1.83µs ± 2%    -7.15%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/otel-bridge.ndjson-12                   4.10µs ± 9%    3.79µs ± 1%    -7.36%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/ratelimit.ndjson-12                     6.87µs ± 7%    6.20µs ± 1%    -9.82%  (p=0.016 n=5+4)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/span-links.ndjson-12                    1.64µs ± 1%    1.57µs ± 1%    -4.13%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/spans.ndjson-12                         11.9µs ± 5%    10.9µs ± 1%    -8.54%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/transactions-huge_traces.ndjson-12      5.86µs ± 3%    5.60µs ± 2%    -4.44%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/transactions.ndjson-12                  11.7µs ± 4%    11.3µs ± 1%    -3.71%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/transactions_spans.ndjson-12            11.5µs ± 2%    11.1µs ± 1%    -3.29%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/transactions_spans_rum.ndjson-12        2.10µs ± 1%    2.05µs ± 1%    -2.52%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/transactions_spans_rum_2.ndjson-12      2.07µs ± 3%    1.98µs ± 1%    -4.31%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/unknown-span-type.ndjson-12             7.60µs ± 4%    7.38µs ± 1%    -2.88%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/errors.ndjson-12                      7.91µs ± 4%    7.41µs ± 2%    -6.41%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/errors_2.ndjson-12                    7.19µs ± 2%    6.92µs ± 1%    -3.72%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/errors_rum.ndjson-12                  2.11µs ±12%    1.99µs ± 1%    -5.49%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/errors_transaction_id.ndjson-12       5.72µs ± 4%    5.48µs ± 2%    -4.13%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/events.ndjson-12                      13.8µs ± 0%    13.2µs ± 2%    -3.91%  (p=0.016 n=4+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/invalid-event-type.ndjson-12           859ns ± 8%     818ns ± 1%    -4.70%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/invalid-event.ndjson-12               3.60µs ± 4%    3.37µs ± 1%    -6.41%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/invalid-json-event.ndjson-12          1.14µs ± 2%    1.10µs ± 1%    -3.84%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/invalid-json-metadata.ndjson-12       2.01µs ± 4%    1.88µs ± 1%    -6.37%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/invalid-metadata-2.ndjson-12           485ns ± 1%     490ns ± 0%    +1.13%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/invalid-metadata.ndjson-12             492ns ± 1%     501ns ± 2%    +1.75%  (p=0.032 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/logs.ndjson-12                        6.10µs ± 5%    5.61µs ± 2%    -7.98%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/metadata-null-values.ndjson-12         841ns ± 4%     807ns ± 1%    -4.12%  (p=0.016 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/metadata.ndjson-12                    1.40µs ±12%    1.29µs ± 2%    -7.97%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/metricsets.ndjson-12                  4.73µs ± 8%    4.41µs ± 2%    -6.88%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/minimal-service.ndjson-12             1.03µs ± 3%    0.99µs ± 2%    -4.07%  (p=0.016 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/optional-timestamps.ndjson-12         1.63µs ± 9%    1.48µs ± 4%    -9.40%  (p=0.016 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/ratelimit.ndjson-12                   5.84µs ± 1%    5.60µs ± 2%    -4.07%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/span-links.ndjson-12                  1.32µs ±13%    1.20µs ± 1%    -8.87%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/spans.ndjson-12                       10.1µs ± 3%     9.5µs ± 3%    -6.63%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/transactions-huge_traces.ndjson-12    4.87µs ± 1%    4.75µs ± 2%    -2.48%  (p=0.032 n=4+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/transactions.ndjson-12                11.7µs ±48%     9.3µs ± 1%   -20.29%  (p=0.016 n=5+4)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/transactions_spans.ndjson-12          10.1µs ± 2%     9.5µs ± 3%    -6.36%  (p=0.016 n=4+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/transactions_spans_rum.ndjson-12      1.77µs ± 3%    1.62µs ± 1%    -8.47%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/transactions_spans_rum_2.ndjson-12    1.67µs ± 5%    1.54µs ± 1%    -8.21%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/unknown-span-type.ndjson-12           6.77µs ± 7%    6.24µs ± 1%    -7.80%  (p=0.008 n=5+5)
ReadBatch/transactions_spans_rum.ndjson-12                                                          16.7µs ±16%    20.1µs ±14%   +20.29%  (p=0.032 n=5+5)
pkg:github.com/elastic/apm-server/internal/publish goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/aggregation/spanmetrics goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/aggregation/txmetrics goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/sampling goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/sampling/eventstorage goos:linux goarch:amd64
WriteTransaction/json_codec-12                                                                      4.61µs ± 9%    9.73µs ±46%  +110.97%  (p=0.008 n=5+5)
WriteTransaction/json_codec_big_tx-12                                                               5.27µs ± 2%    8.37µs ±54%   +58.88%  (p=0.016 n=4+5)
ReadEvents/json_codec_big_tx/0_events-12                                                             303ns ± 1%     344ns ± 6%   +13.37%  (p=0.008 n=5+5)
ReadEvents/nop_codec/399_events-12                                                                   418µs ± 9%     472µs ±11%   +12.86%  (p=0.016 n=5+5)
ReadEvents/nop_codec_big_tx/0_events-12                                                              318ns ± 4%     336ns ± 4%    +5.61%  (p=0.032 n=5+5)
IsTraceSampled/unknown-12                                                                            380ns ± 5%     409ns ± 5%    +7.43%  (p=0.016 n=5+5)

name                                                                                              old alloc/op   new alloc/op   delta
pkg:github.com/elastic/apm-server/internal/agentcfg goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/beater/request goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/model/modelindexer goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/processor/stream goos:linux goarch:amd64
BackendProcessor/invalid-event-type.ndjson-12                                                       3.74kB ± 1%    3.67kB ± 1%    -1.89%  (p=0.008 n=5+5)
BackendProcessor/transactions_spans_rum.ndjson-12                                                   5.83kB ± 2%    5.74kB ± 1%    -1.57%  (p=0.032 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel2/otel-bridge.ndjson-12                   11.7kB ± 1%    11.9kB ± 1%    +1.54%  (p=0.024 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel2/spans.ndjson-12                         30.6kB ± 1%    30.3kB ± 1%    -0.85%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel4/logs.ndjson-12                          20.8kB ± 1%    21.1kB ± 1%    +1.24%  (p=0.032 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/spans.ndjson-12                       29.6kB ± 1%    29.9kB ± 1%    +0.85%  (p=0.032 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/transactions_spans_rum.ndjson-12      6.09kB ± 1%    6.25kB ± 2%    +2.71%  (p=0.016 n=5+5)
ReadBatch/errors_2.ndjson-12                                                                        22.7kB ± 0%    22.7kB ± 0%    -0.06%  (p=0.008 n=5+5)
ReadBatch/invalid-json-metadata.ndjson-12                                                           8.42kB ± 0%    8.38kB ± 0%    -0.52%  (p=0.032 n=5+5)
ReadBatch/logs.ndjson-12                                                                            18.4kB ± 0%    18.4kB ± 0%    +0.15%  (p=0.008 n=5+5)
ReadBatch/metadata-null-values.ndjson-12                                                            2.42kB ± 0%    2.42kB ± 0%    +0.06%  (p=0.032 n=5+5)
ReadBatch/transactions_spans.ndjson-12                                                              22.8kB ± 0%    22.8kB ± 0%    -0.07%  (p=0.008 n=5+5)
pkg:github.com/elastic/apm-server/internal/publish goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/aggregation/spanmetrics goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/aggregation/txmetrics goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/sampling goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/sampling/eventstorage goos:linux goarch:amd64
ReadEvents/nop_codec/199_events-12                                                                   470kB ± 0%     470kB ± 0%    -0.17%  (p=0.016 n=5+4)

name                                                                                              old allocs/op  new allocs/op  delta
pkg:github.com/elastic/apm-server/internal/agentcfg goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/beater/request goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/model/modelindexer goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/processor/stream goos:linux goarch:amd64
BackendProcessor/heavy.ndjson-12                                                                     22.3k ± 0%     22.3k ± 0%    -0.00%  (p=0.016 n=5+4)
BackendProcessor/logs.ndjson-12                                                                        332 ± 0%       333 ± 0%    +0.30%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel2/logs.ndjson-12                             332 ± 0%       333 ± 0%    +0.30%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel4/logs.ndjson-12                             332 ± 0%       333 ± 0%    +0.30%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/logs.ndjson-12                             332 ± 0%       333 ± 0%    +0.30%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/logs.ndjson-12                           332 ± 0%       333 ± 0%    +0.30%  (p=0.008 n=5+5)
ReadBatch/logs.ndjson-12                                                                               329 ± 0%       330 ± 0%    +0.30%  (p=0.008 n=5+5)
pkg:github.com/elastic/apm-server/internal/publish goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/aggregation/spanmetrics goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/aggregation/txmetrics goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/sampling goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/sampling/eventstorage goos:linux goarch:amd64

name                                                                                              old speed      new speed      delta
pkg:github.com/elastic/apm-server/internal/model/modelindexer goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/processor/stream goos:linux goarch:amd64
BackendProcessor/metadata-null-values.ndjson-12                                                   34.9MB/s ± 9%  46.1MB/s ±44%   +32.26%  (p=0.016 n=4+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel4/otel-bridge.ndjson-12                  269MB/s ± 8%   244MB/s ± 5%    -9.23%  (p=0.032 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel4/spans.ndjson-12                        414MB/s ± 3%   374MB/s ±10%    -9.82%  (p=0.032 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/events.ndjson-12                       469MB/s ± 5%   486MB/s ± 1%    +3.70%  (p=0.032 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/metadata-null-values.ndjson-12         445MB/s ± 1%   456MB/s ± 1%    +2.46%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/minimal-service.ndjson-12              301MB/s ± 4%   316MB/s ± 1%    +4.80%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/optional-timestamps.ndjson-12          522MB/s ± 4%   562MB/s ± 2%    +7.66%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/otel-bridge.ndjson-12                  460MB/s ± 9%   496MB/s ± 1%    +7.72%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/ratelimit.ndjson-12                    614MB/s ± 7%   679MB/s ± 1%   +10.72%  (p=0.016 n=5+4)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/span-links.ndjson-12                   416MB/s ± 1%   433MB/s ± 1%    +4.31%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/spans.ndjson-12                        675MB/s ± 5%   737MB/s ± 1%    +9.20%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/transactions-huge_traces.ndjson-12     540MB/s ± 3%   565MB/s ± 2%    +4.62%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/transactions.ndjson-12                 483MB/s ± 4%   501MB/s ± 1%    +3.80%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/transactions_spans.ndjson-12           505MB/s ± 2%   522MB/s ± 1%    +3.39%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/transactions_spans_rum.ndjson-12       550MB/s ± 1%   565MB/s ± 1%    +2.60%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/transactions_spans_rum_2.ndjson-12     539MB/s ± 3%   563MB/s ± 1%    +4.48%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/unknown-span-type.ndjson-12            435MB/s ± 4%   448MB/s ± 1%    +2.93%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/errors.ndjson-12                     802MB/s ± 4%   857MB/s ± 2%    +6.79%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/errors_2.ndjson-12                   656MB/s ± 2%   681MB/s ± 1%    +3.83%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/errors_rum.ndjson-12                 904MB/s ±11%   953MB/s ± 1%    +5.42%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/errors_transaction_id.ndjson-12      669MB/s ± 4%   697MB/s ± 2%    +4.26%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/events.ndjson-12                     539MB/s ± 0%   561MB/s ± 2%    +4.09%  (p=0.016 n=4+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/invalid-event-type.ndjson-12         456MB/s ± 8%   478MB/s ± 1%    +4.77%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/invalid-event.ndjson-12              213MB/s ± 4%   228MB/s ± 1%    +6.79%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/invalid-json-event.ndjson-12         515MB/s ± 2%   535MB/s ± 1%    +3.97%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/invalid-json-metadata.ndjson-12      223MB/s ± 4%   238MB/s ± 1%    +6.75%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/invalid-metadata-2.ndjson-12         900MB/s ± 1%   890MB/s ± 0%    -1.12%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/invalid-metadata.ndjson-12           907MB/s ± 1%   891MB/s ± 2%    -1.72%  (p=0.032 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/logs.ndjson-12                       573MB/s ± 5%   625MB/s ± 2%    +9.12%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/metadata-null-values.ndjson-12       626MB/s ± 4%   652MB/s ± 1%    +4.22%  (p=0.016 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/metadata.ndjson-12                   890MB/s ±11%   963MB/s ± 2%    +8.22%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/metricsets.ndjson-12                 539MB/s ± 7%   578MB/s ± 2%    +7.23%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/minimal-service.ndjson-12            414MB/s ± 3%   431MB/s ± 2%    +4.23%  (p=0.016 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/optional-timestamps.ndjson-12        630MB/s ± 9%   694MB/s ± 4%   +10.08%  (p=0.016 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/ratelimit.ndjson-12                  722MB/s ± 1%   753MB/s ± 2%    +4.26%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/span-links.ndjson-12                 519MB/s ±12%   566MB/s ± 1%    +9.17%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/spans.ndjson-12                      792MB/s ± 3%   848MB/s ± 3%    +7.09%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/transactions-huge_traces.ndjson-12   651MB/s ± 1%   668MB/s ± 2%    +2.57%  (p=0.032 n=4+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/transactions_spans.ndjson-12         574MB/s ± 2%   613MB/s ± 3%    +6.79%  (p=0.016 n=4+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/transactions_spans_rum.ndjson-12     651MB/s ± 3%   711MB/s ± 1%    +9.26%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/transactions_spans_rum_2.ndjson-12   668MB/s ± 5%   727MB/s ± 1%    +8.78%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/unknown-span-type.ndjson-12          489MB/s ± 7%   530MB/s ± 1%    +8.28%  (p=0.008 n=5+5)
ReadBatch/transactions_spans_rum.ndjson-12                                                        70.2MB/s ±17%  57.9MB/s ±13%   -17.54%  (p=0.032 n=5+5)

report generated with https://pkg.go.dev/golang.org/x/perf/cmd/benchstat

@axw axw merged commit a27b902 into elastic:main Nov 24, 2022
@mergify mergify bot mentioned this pull request Nov 24, 2022
Merged
mergify bot pushed a commit that referenced this pull request Nov 24, 2022
@axw @mergify
Logs intake fixes (#9667)
949eee7 
* modeldecoder: accept "event.dataset"

The intake accepts the ECS field, "event.dataset",
and copies it to the field of the same name in
resulting documents.

* Only set error.grouping_key for error events

Check the processor.* value, don't just take the
presence of error.* fields to mean the event is
an error.

* Add logs intake fields to app_logs data stream

(cherry picked from commit a27b902)
axw added a commit that referenced this pull request Nov 24, 2022
@mergify @axw
Logs intake fixes (#9667) (#9671)
ecadb11 
* modeldecoder: accept "event.dataset"

The intake accepts the ECS field, "event.dataset",
and copies it to the field of the same name in
resulting documents.

* Only set error.grouping_key for error events

Check the processor.* value, don't just take the
presence of error.* fields to mean the event is
an error.

* Add logs intake fields to app_logs data stream

(cherry picked from commit a27b902)

Co-authored-by: Andrew Wilkins <[email protected]>
@simitt simitt self-assigned this Dec 5, 2022
@simitt
Copy link
Contributor

simitt commented Dec 5, 2022

Tested with BC5 on ESS - the fields added by this PR are not yet part of the index template. This might be related to the bug that the apmpackage has not been updated in the last days.

The error.grouping_key field was empty as expected.

@axw axw deleted the logs-fixes branch December 6, 2022 01:22
@elasticmachine
Copy link
Contributor

Package apm - 8.6.0-preview-1670294014 containing this change is available at https://epr.elastic.co/search?package=apm

@simitt simitt removed their assignment Dec 6, 2022
@kruskall kruskall self-assigned this Dec 7, 2022
@kruskall
Copy link
Member

Created a deployment and sent the json payload in the linked PR, created a data view for the logs* index. Running a error.grouping_key : * query in kibana shows no results and all the fields seem to be mapped correctly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lahsivjar lahsivjar lahsivjar approved these changes

Assignees

@kruskall kruskall

Labels
backport-8.6 Automated backport with mergify test-plan test-plan-ok v8.6.0
Projects
None yet
Milestone
No milestone
6 participants
@axw @apmmachine @simitt @elasticmachine @kruskall @lahsivjar