testing/rally: add force-merge operation

[axw]

Motivation/summary

Force merge shards to ensure storage size metrics are consistently captured after shards are merged.

While testing #9215, I found that the storage size would reduce after a short time -- because of a delayed reaction segment merge.

Checklist

- [ ] Update CHANGELOG.asciidoc
- [ ] Update package changelog.yml (only if changes to apmpackage have been made)
- [ ] Documentation has been updated

How to test these changes

N/A

Related issues

None

[mergify]

This pull request does not have a backport label. Could you fix it @axw? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-7.x is the label to automatically backport to the 7.x branch.
• backport-7./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

[apmmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-09-29T01:04:41.639+0000

• Duration: 29 min 47 sec

Test stats 🧪

Test	Results
Failed	0
Passed	149
Skipped	0
Total	149

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate and publish the docker images.

• /test windows : Build & tests on Windows.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[apmmachine]

📚 Go benchmark report

Diff with the main branch

name                                                                                              old time/op    new time/op     delta
pkg:github.com/elastic/apm-server/internal/agentcfg goos:linux goarch:amd64
FetchAndAdd/FetchFromCache-12                                                                       46.1ns ± 0%     41.2ns ± 0%  -10.74%  (p=0.008 n=5+5)
FetchAndAdd/FetchAndAddToCache-12                                                                    107ns ± 5%       96ns ± 3%  -10.71%  (p=0.008 n=5+5)
pkg:github.com/elastic/apm-server/internal/beater/request goos:linux goarch:amd64
ContextResetContentEncoding/empty-12                                                                 132ns ± 1%      118ns ± 1%  -10.19%  (p=0.008 n=5+5)
ContextResetContentEncoding/uncompressed-12                                                          158ns ± 1%      142ns ± 1%   -9.79%  (p=0.008 n=5+5)
pkg:github.com/elastic/apm-server/internal/model/modelindexer goos:linux goarch:amd64
ModelIndexer/DefaultCompression-12                                                                  9.45µs ±31%     5.87µs ±15%  -37.83%  (p=0.008 n=5+5)
pkg:github.com/elastic/apm-server/internal/processor/stream goos:linux goarch:amd64
BackendProcessor/errors_transaction_id.ndjson-12                                                    56.1µs ±33%     76.8µs ±14%  +36.87%  (p=0.032 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel2/errors_transaction_id.ndjson-12         22.0µs ±23%     18.4µs ± 8%  -16.52%  (p=0.016 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel2/transactions.ndjson-12                  36.1µs ± 8%     31.7µs ±12%  -12.17%  (p=0.032 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel4/optional-timestamps.ndjson-12           3.52µs ± 7%     3.01µs ±14%  -14.59%  (p=0.016 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/invalid-metadata.ndjson-12               838ns ± 1%      850ns ± 1%   +1.43%  (p=0.032 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/transactions-huge_traces.ndjson-12      5.59µs ± 1%     5.47µs ± 1%   -2.16%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/transactions_spans.ndjson-12            11.1µs ± 1%     10.7µs ± 1%   -3.28%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/transactions_spans_rum.ndjson-12        2.04µs ± 1%     1.99µs ± 1%   -2.38%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/transactions_spans_rum_2.ndjson-12      1.97µs ± 1%     1.93µs ± 1%   -2.16%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/unknown-span-type.ndjson-12             7.33µs ± 0%     7.12µs ± 2%   -2.86%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/errors.ndjson-12                      7.41µs ± 1%     6.89µs ± 4%   -6.98%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/errors_2.ndjson-12                    6.96µs ± 4%     6.59µs ± 4%   -5.39%  (p=0.032 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/errors_transaction_id.ndjson-12       5.45µs ± 0%     5.40µs ± 1%   -0.89%  (p=0.016 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/heavy.ndjson-12                        516µs ± 1%      502µs ± 2%   -2.53%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/invalid-event-type.ndjson-12           799ns ± 1%      785ns ± 1%   -1.73%  (p=0.032 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/invalid-json-metadata.ndjson-12       1.89µs ± 1%     1.87µs ± 0%   -1.27%  (p=0.016 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/invalid-metadata-2.ndjson-12           472ns ± 1%      464ns ± 1%   -1.75%  (p=0.016 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/logs.ndjson-12                        1.91µs ± 1%     1.85µs ± 3%   -2.84%  (p=0.032 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/optional-timestamps.ndjson-12         1.47µs ± 2%     1.42µs ± 2%   -3.68%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/transactions-huge_traces.ndjson-12    4.71µs ± 2%     4.53µs ± 1%   -3.77%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/transactions.ndjson-12                9.35µs ± 1%     9.17µs ± 1%   -1.94%  (p=0.016 n=5+5)
ReadBatch/transactions.ndjson-12                                                                    93.5µs ±18%    122.0µs ±21%  +30.54%  (p=0.016 n=5+5)
pkg:github.com/elastic/apm-server/internal/publish goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/aggregation/spanmetrics goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/aggregation/txmetrics goos:linux goarch:amd64
AggregateTransaction-12                                                                             83.3ns ± 1%     82.3ns ± 1%   -1.17%  (p=0.032 n=5+5)
pkg:github.com/elastic/apm-server/x-pack/apm-server/sampling goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/sampling/eventstorage goos:linux goarch:amd64
WriteTransaction/json_codec-12                                                                      9.61µs ±54%     4.10µs ± 2%  -57.38%  (p=0.008 n=5+5)
WriteTransaction/json_codec_big_tx-12                                                               14.9µs ±25%      5.2µs ±26%  -65.20%  (p=0.008 n=5+5)
ReadEvents/json_codec/0_events-12                                                                    347ns ± 5%      306ns ± 5%  -11.96%  (p=0.008 n=5+5)
ReadEvents/nop_codec/0_events-12                                                                     338ns ± 5%      311ns ± 8%   -7.76%  (p=0.032 n=5+5)
ReadEvents/nop_codec_big_tx/0_events-12                                                              340ns ± 6%      301ns ± 5%  -11.65%  (p=0.008 n=5+5)
IsTraceSampled/sampled-12                                                                           74.4ns ± 3%     68.2ns ± 6%   -8.33%  (p=0.008 n=5+5)
IsTraceSampled/unsampled-12                                                                         77.2ns ± 1%     69.4ns ± 1%  -10.16%  (p=0.008 n=5+5)
IsTraceSampled/unknown-12                                                                            420ns ± 2%      373ns ± 3%  -11.15%  (p=0.008 n=5+5)

name                                                                                              old alloc/op   new alloc/op    delta
pkg:github.com/elastic/apm-server/internal/agentcfg goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/beater/request goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/model/modelindexer goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/processor/stream goos:linux goarch:amd64
BackendProcessor/invalid-metadata-2.ndjson-12                                                       2.70kB ± 1%     2.75kB ± 2%   +2.11%  (p=0.032 n=5+5)
BackendProcessor/metadata-null-values.ndjson-12                                                     3.25kB ± 1%     3.33kB ± 1%   +2.26%  (p=0.008 n=5+5)
BackendProcessor/metricsets.ndjson-12                                                               13.1kB ± 1%     13.3kB ± 0%   +1.99%  (p=0.016 n=5+4)
BackendProcessor/optional-timestamps.ndjson-12                                                      4.46kB ± 1%     4.57kB ± 1%   +2.56%  (p=0.008 n=5+5)
BackendProcessor/otel-bridge.ndjson-12                                                              10.9kB ± 1%     11.0kB ± 0%   +1.08%  (p=0.016 n=5+5)
BackendProcessor/transactions_spans_rum_2.ndjson-12                                                 5.44kB ± 2%     5.57kB ± 2%   +2.36%  (p=0.032 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel2/metricsets.ndjson-12                    14.4kB ± 1%     14.4kB ± 0%   +0.61%  (p=0.032 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel2/unknown-span-type.ndjson-12             18.6kB ± 0%     18.9kB ± 1%   +1.40%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel4/metadata-null-values.ndjson-12          3.79kB ± 1%     3.73kB ± 1%   -1.54%  (p=0.016 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel4/unknown-span-type.ndjson-12             19.0kB ± 1%     18.7kB ± 1%   -1.60%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/events.ndjson-12                        41.8kB ± 1%     42.2kB ± 1%   +0.79%  (p=0.032 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/invalid-json-event.ndjson-12            4.72kB ± 2%     4.79kB ± 1%   +1.51%  (p=0.040 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/transactions.ndjson-12                  27.5kB ± 1%     27.7kB ± 0%   +0.78%  (p=0.032 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/unknown-span-type.ndjson-12           18.4kB ± 1%     18.3kB ± 0%   -0.65%  (p=0.032 n=5+5)
ReadBatch/errors_2.ndjson-12                                                                        22.6kB ± 0%     22.6kB ± 0%   +0.14%  (p=0.008 n=5+5)
ReadBatch/events.ndjson-12                                                                          38.6kB ± 0%     38.7kB ± 0%   +0.29%  (p=0.032 n=5+5)
ReadBatch/invalid-metadata-2.ndjson-12                                                              5.80kB ± 0%     5.80kB ± 0%   +0.04%  (p=0.048 n=5+5)
ReadBatch/metricsets.ndjson-12                                                                      15.5kB ± 0%     15.6kB ± 0%   +0.47%  (p=0.016 n=5+5)
ReadBatch/minimal-service.ndjson-12                                                                 2.76kB ± 0%     2.76kB ± 0%   +0.06%  (p=0.048 n=5+5)
ReadBatch/transactions-huge_traces.ndjson-12                                                        13.8kB ± 0%     13.8kB ± 0%   +0.03%  (p=0.040 n=5+5)
ReadBatch/transactions_spans.ndjson-12                                                              22.8kB ± 0%     22.8kB ± 0%   +0.05%  (p=0.024 n=5+5)
pkg:github.com/elastic/apm-server/internal/publish goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/aggregation/spanmetrics goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/aggregation/txmetrics goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/sampling goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/sampling/eventstorage goos:linux goarch:amd64
WriteTransaction/json_codec_big_tx-12                                                               3.90kB ± 0%     3.90kB ± 0%   -0.04%  (p=0.000 n=4+5)
ReadEvents/nop_codec/10_events-12                                                                   25.7kB ± 0%     25.7kB ± 0%   +0.06%  (p=0.048 n=5+5)
ReadEvents/nop_codec_big_tx/100_events-12                                                            250kB ± 0%      250kB ± 0%   +0.08%  (p=0.016 n=5+5)

name                                                                                              old allocs/op  new allocs/op   delta
pkg:github.com/elastic/apm-server/internal/agentcfg goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/beater/request goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/model/modelindexer goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/processor/stream goos:linux goarch:amd64
BackendProcessor/events.ndjson-12                                                                      716 ± 0%        717 ± 0%   +0.14%  (p=0.016 n=4+5)
pkg:github.com/elastic/apm-server/internal/publish goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/aggregation/spanmetrics goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/aggregation/txmetrics goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/sampling goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/sampling/eventstorage goos:linux goarch:amd64
ReadEvents/json_codec_big_tx/1000_events-12                                                          60.5k ± 0%      60.5k ± 0%   -0.00%  (p=0.029 n=4+4)

name                                                                                              old speed      new speed       delta
pkg:github.com/elastic/apm-server/internal/processor/stream goos:linux goarch:amd64
BackendProcessor/errors_transaction_id.ndjson-12                                                  72.8MB/s ±41%   50.1MB/s ±15%  -31.16%  (p=0.032 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel2/errors_transaction_id.ndjson-12        176MB/s ±20%    209MB/s ± 8%  +18.40%  (p=0.016 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel2/transactions.ndjson-12                 157MB/s ± 8%    179MB/s ±11%  +14.22%  (p=0.032 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel4/optional-timestamps.ndjson-12          292MB/s ± 7%    343MB/s ±12%  +17.41%  (p=0.016 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/invalid-metadata.ndjson-12             532MB/s ± 1%    525MB/s ± 1%   -1.41%  (p=0.032 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/transactions-huge_traces.ndjson-12     568MB/s ± 1%    580MB/s ± 1%   +2.21%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/transactions_spans.ndjson-12           524MB/s ± 1%    542MB/s ± 1%   +3.39%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/transactions_spans_rum.ndjson-12       567MB/s ± 1%    581MB/s ± 1%   +2.45%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/transactions_spans_rum_2.ndjson-12     568MB/s ± 1%    580MB/s ± 1%   +2.20%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/unknown-span-type.ndjson-12            451MB/s ± 0%    465MB/s ± 2%   +2.96%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/errors.ndjson-12                     857MB/s ± 1%    921MB/s ± 4%   +7.56%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/errors_2.ndjson-12                   677MB/s ± 4%    716MB/s ± 4%   +5.71%  (p=0.032 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/errors_transaction_id.ndjson-12      701MB/s ± 0%    708MB/s ± 1%   +0.90%  (p=0.016 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/heavy.ndjson-12                      774MB/s ± 1%    794MB/s ± 2%   +2.60%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/invalid-event-type.ndjson-12         489MB/s ± 1%    498MB/s ± 1%   +1.77%  (p=0.032 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/invalid-json-metadata.ndjson-12      236MB/s ± 1%    239MB/s ± 0%   +1.26%  (p=0.016 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/invalid-metadata-2.ndjson-12         924MB/s ± 1%    941MB/s ± 1%   +1.79%  (p=0.016 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/logs.ndjson-12                       695MB/s ± 1%    716MB/s ± 4%   +2.95%  (p=0.032 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/optional-timestamps.ndjson-12        697MB/s ± 2%    724MB/s ± 2%   +3.82%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/transactions-huge_traces.ndjson-12   674MB/s ± 2%    700MB/s ± 1%   +3.90%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/transactions.ndjson-12               604MB/s ± 1%    616MB/s ± 1%   +1.99%  (p=0.016 n=5+5)
ReadBatch/transactions.ndjson-12                                                                  61.2MB/s ±20%   46.9MB/s ±19%  -23.36%  (p=0.016 n=5+5)

report generated with https://pkg.go.dev/golang.org/x/perf/cmd/benchstat