filebeat benchmark input and discard output

[leehinman]

Proposed commit message

filebeat benchmark input and discard output

benchmark is simple input that generates synthetic events, useful for benchmarking outputs or for providing load on outputs.

The message of each event is set in the config file. Event duplication is avoided with the thread, line and filename fields. Each go routine that generates an event sets a unique thread field. Within each each thread, every event that is generated increments the line field. The line field is a uint64, if the max is reached then the filename field is incremented and line is reset to 0.

The default behavior is to generate events as quickly as possible until filebeat is stopped. Optionally you can set the count option, which will generate that number of events and then stop. Or you can use the eps option which allows you to set an events generated per second.

discard is a simple output that throws the data away, but records in the metrics that an event was successfully sent.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

• [ ]

How to test this PR locally

In filebeat.yml add:

filebeat.inputs:
- type: benchmark
  message: "uninspired test message"

output.discard:

This should result in events with the message field set to "uninspried test message", generated as quickly as possible until filebeat is stopped.

Related issues

Use cases

• tracking down performance issues
• generating a consistent load for outputs
• performance testing inputs and processors

Screenshots

Logs

{
  "@timestamp": "2023-12-05T19:09:49.504Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "8.12.0"
  },
  "ecs": {
    "version": "8.0.0"
  },
  "host": {
    "name": "elastic2"
  },
  "agent": {
    "id": "796fb7fc-e337-4ee0-b1f2-88f133a5e801",
    "name": "elastic2",
    "type": "filebeat",
    "version": "8.12.0",
    "ephemeral_id": "d3318883-aac0-4903-a1fc-1692aade498f"
  },
  "message": "generic message",
  "line": 21049584,
  "filename": 0,
  "thread": 0,
  "input": {
    "type": "benchmark"
  }
}

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @leehinman? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

[elasticmachine]

❕ Build Aborted

Either there was a build timeout or someone aborted the build.

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Duration: 7 min 50 sec

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[elasticmachine]

❕ Build Aborted

Either there was a build timeout or someone aborted the build.

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Duration: 31 min 21 sec

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2024-04-11T19:56:25.539+0000

• Duration: 150 min 3 sec

Test stats 🧪

Test	Results
Failed	0
Passed	29452
Skipped	2061
Total	31513

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b spacetime_bench_input upstream/spacetime_bench_input
git merge upstream/main
git push upstream spacetime_bench_input

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b spacetime_bench_input upstream/spacetime_bench_input
git merge upstream/main
git push upstream spacetime_bench_input

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @leehinman? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b spacetime_bench_input upstream/spacetime_bench_input
git merge upstream/main
git push upstream spacetime_bench_input

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 629c19e

History

• 💚 Build #2556 succeeded 436351a
• 💚 Build #2555 succeeded e8f8b7b
• 💚 Build #2033 succeeded 454f115
• 💚 Build #1904 succeeded b63732a
• 💚 Build #1886 succeeded c93621f

cc @leehinman

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 629c19e

History

• 💚 Build #1322 succeeded 436351a
• 💚 Build #1321 succeeded e8f8b7b

cc @leehinman

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 629c19e

History

• 💚 Build #1280 succeeded 436351a
• 💚 Build #1279 succeeded e8f8b7b

cc @leehinman

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 629c19e

History

• 💚 Build #1715 succeeded 436351a
• 💚 Build #1714 succeeded e8f8b7b
• 💚 Build #1196 succeeded 454f115
• 💚 Build #1067 succeeded b63732a
• 💚 Build #1049 succeeded c93621f

cc @leehinman

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 629c19e

History

• 💚 Build #2559 succeeded 436351a
• 💚 Build #2558 succeeded e8f8b7b
• 💔 Build #2040 failed 454f115
• 💔 Build #1911 failed b63732a
• 💚 Build #1893 succeeded c93621f

cc @leehinman

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 629c19e

History

• 💚 Build #1277 succeeded 436351a
• 💚 Build #1276 succeeded e8f8b7b

cc @leehinman

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 629c19e

History

• 💚 Build #2849 succeeded 436351a
• 💚 Build #2848 succeeded e8f8b7b
• 💔 Build #2330 failed 454f115
• 💔 Build #2201 failed b63732a
• 💔 Build #2182 failed c93621f

cc @leehinman

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 629c19e

History

• 💚 Build #1290 succeeded 436351a
• 💚 Build #1289 succeeded e8f8b7b

cc @leehinman

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 629c19e

History

• 💔 Build #3770 failed 436351a
• 💚 Build #3769 succeeded e8f8b7b
• 💔 Build #3247 failed 454f115
• 💚 Build #3118 succeeded b63732a
• 💚 Build #3100 succeeded c93621f

cc @leehinman

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 629c19e

History

• 💚 Build #4205 succeeded 436351a
• 💔 Build #4204 failed e8f8b7b
• 💔 Build #3671 failed 454f115
• 💔 Build #3542 failed b63732a
• 💔 Build #3523 failed c93621f

cc @leehinman