[Filebeat] Disable harvester detailed metrics by default #22555

andrewstucki · 2020-11-11T20:45:09Z

What does this PR do?

This disables the detailed filebeat harvester metrics added in #13395 by default. In order to enable them, you need to set a detailed_metrics configuration option to true.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

Closes filebeat creates mapping explosion with field monitoring.metrics.filebeat.harvester.files #19745

elasticmachine · 2020-11-11T20:46:01Z

Pinging @elastic/integrations-services (Team:Services)

elasticmachine · 2020-11-11T20:46:01Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

elasticmachine · 2020-11-11T21:06:22Z

❕ Build Aborted

There is a new build on-going so the previous on-going builds have been aborted.

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: [Pull request #22555 opened]
Reason: Aborted from #2
Start Time: 2020-11-11T20:45:52.333+0000
Duration: 20 min 19 sec
Commit: 82476dd

Test stats 🧪

Test	Results
Failed	0
Passed	1415
Skipped	264
Total	1679

Steps errors

Expand to view the steps failures

`Error signal`

Took 0 min 0 sec . View more details on here
Description: Error 'org.jenkinsci.plugins.workflow.steps.FlowInterruptedException'

`Error signal`

Took 0 min 0 sec . View more details on here
Description: Error 'org.jenkinsci.plugins.workflow.steps.FlowInterruptedException'

`x-pack/filebeat-Lint - make -C x-pack/filebeat check;

make -C x-pack/filebeat update;
make check-no-`

Took 3 min 48 sec . View more details on here
Description: make -C x-pack/filebeat check;make -C x-pack/filebeat update;make check-no-changes;

`Error signal`

Took 0 min 0 sec . View more details on here
Description: Error 'hudson.AbortException: script returned exit code 2'

`Error signal`

Took 0 min 0 sec . View more details on here
Description: Error 'org.jenkinsci.plugins.workflow.steps.FlowInterruptedException'

`Error signal`

Took 0 min 0 sec . View more details on here
Description: Error 'org.jenkinsci.plugins.workflow.steps.FlowInterruptedException'

Log output

Expand to view the last 100 lines of log output

[2020-11-11T21:05:13.154Z]   OS/Arch:          linux/amd64
[2020-11-11T21:05:13.154Z]   Experimental:     false
[2020-11-11T21:05:13.154Z]  containerd:
[2020-11-11T21:05:13.154Z]   Version:          1.3.7
[2020-11-11T21:05:13.154Z]   GitCommit:        8fba4e9a7d01810a393d5d25a3621dc101981175
[2020-11-11T21:05:13.154Z]  runc:
[2020-11-11T21:05:13.154Z]   Version:          1.0.0-rc10
[2020-11-11T21:05:13.154Z]   GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
[2020-11-11T21:05:13.154Z]  docker-init:
[2020-11-11T21:05:13.154Z]   Version:          0.18.0
[2020-11-11T21:05:13.154Z]   GitCommit:        fec3683
[2020-11-11T21:05:13.550Z] Client: Docker Engine - Community
[2020-11-11T21:05:13.550Z]  Version:           19.03.13
[2020-11-11T21:05:13.550Z]  API version:       1.40
[2020-11-11T21:05:13.550Z]  Go version:        go1.13.15
[2020-11-11T21:05:13.550Z]  Git commit:        4484c46d9d
[2020-11-11T21:05:13.550Z]  Built:             Wed Sep 16 17:02:36 2020
[2020-11-11T21:05:13.550Z]  OS/Arch:           linux/amd64
[2020-11-11T21:05:13.550Z]  Experimental:      false
[2020-11-11T21:05:13.550Z] 
[2020-11-11T21:05:13.550Z] Server: Docker Engine - Community
[2020-11-11T21:05:13.550Z]  Engine:
[2020-11-11T21:05:13.550Z]   Version:          19.03.13
[2020-11-11T21:05:13.550Z]   API version:      1.40 (minimum version 1.12)
[2020-11-11T21:05:13.550Z]   Go version:       go1.13.15
[2020-11-11T21:05:13.550Z]   Git commit:       4484c46d9d
[2020-11-11T21:05:13.550Z]   Built:            Wed Sep 16 17:01:06 2020
[2020-11-11T21:05:13.550Z]   OS/Arch:          linux/amd64
[2020-11-11T21:05:13.550Z]   Experimental:     false
[2020-11-11T21:05:13.550Z]  containerd:
[2020-11-11T21:05:13.550Z]   Version:          1.3.7
[2020-11-11T21:05:13.550Z]   GitCommit:        8fba4e9a7d01810a393d5d25a3621dc101981175
[2020-11-11T21:05:13.550Z]  runc:
[2020-11-11T21:05:13.550Z]   Version:          1.0.0-rc10
[2020-11-11T21:05:13.550Z]   GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
[2020-11-11T21:05:13.550Z]  docker-init:
[2020-11-11T21:05:13.550Z]   Version:          0.18.0
[2020-11-11T21:05:13.550Z]   GitCommit:        fec3683
[2020-11-11T21:05:12.794Z] [Google Cloud Storage Plugin] Uploading: x-pack-filebeat--system-tests-linux.tgz
[2020-11-11T21:05:14.616Z] Failed in branch x-pack/filebeat-windows-windows-2019
[2020-11-11T21:05:18.746Z] Failed in branch x-pack/filebeat-build
[2020-11-11T21:05:19.146Z] Failed in branch filebeat-build
[2020-11-11T21:05:19.464Z] Stage "Packaging" skipped due to earlier failure(s)
[2020-11-11T21:05:24.138Z] [INFO] unstashV2: JOB_GCS_BUCKET is set. bucket param got precedency instead.
[2020-11-11T21:05:24.174Z] [INFO] unstashV2: JOB_GCS_CREDENTIALS is set. credentialsId param got precedency instead.
[2020-11-11T21:05:24.233Z] [Google Cloud Storage Plugin] Found 1 files to download from pattern: gs://beats-ci-temp/Beats/beats/PR-22555-1/source/source.tgz
[2020-11-11T21:05:24.255Z] [Google Cloud Storage Plugin] Downloading: Beats/beats/PR-22555-1/source/source.tgz to local path: /var/lib/jenkins/workspace/Beats_beats_PR-22555/source.tgz
[2020-11-11T21:05:35.707Z] + tar --version
[2020-11-11T21:05:36.030Z] + tar -xpf source.tgz
[2020-11-11T21:05:48.587Z] + rm source.tgz
[2020-11-11T21:05:48.684Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22555/src/github.com/elastic/beats
[2020-11-11T21:05:48.716Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22555/src/github.com/elastic/beats/filebeat-Lint-1605128472204
[2020-11-11T21:05:48.821Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22555/src/github.com/elastic/beats/x-pack-filebeat-Lint-1605128512681
[2020-11-11T21:05:48.917Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22555/src/github.com/elastic/beats/filebeat-windows-windows-2019-1605128694884
[2020-11-11T21:05:49.011Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22555/src/github.com/elastic/beats/x-pack-filebeat-windows-windows-2019-1605128696422
[2020-11-11T21:05:49.112Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22555/src/github.com/elastic/beats/filebeat-build-1605128702145
[2020-11-11T21:05:49.213Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22555/src/github.com/elastic/beats/x-pack-filebeat-build-1605128703213
[2020-11-11T21:05:49.668Z] + cat
[2020-11-11T21:05:49.668Z] + /usr/local/bin/runbld ./runbld-test-reports --job-name elastic+beats+pull-request
[2020-11-11T21:05:49.668Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-11-11T21:05:56.245Z] runbld>>> runbld started
[2020-11-11T21:05:56.245Z] runbld>>> 1.6.12/f45d832f2ba0aa2722ab4ec1fda8ad140f027f8b
[2020-11-11T21:05:57.188Z] runbld>>> The following profiles matched the job 'elastic+beats+pull-request' in order of occurrence in the config (last value wins).
[2020-11-11T21:05:57.188Z] runbld>>> Matches in the system config:
[2020-11-11T21:05:57.188Z] runbld>>> - Matched ^elastic\+beats
[2020-11-11T21:05:57.188Z] runbld>>> - Matched ^elastic\+beats\+pull-request
[2020-11-11T21:05:58.565Z] runbld>>> Debug logging enabled.
[2020-11-11T21:05:58.565Z] runbld>>> Storing result
[2020-11-11T21:05:58.565Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-11-11T21:05:58.565Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1597739501209/t/20201111210558-24C29AB2
[2020-11-11T21:05:58.565Z] runbld>>> Adding system facts.
[2020-11-11T21:05:59.951Z] runbld>>> Adding vcs info for the latest commit:  82476dd7896d152108b00c7ed8312bbb03d82eee
[2020-11-11T21:05:59.951Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-11-11T21:05:59.951Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-11-11T21:05:59.951Z] + echo 'Processing JUnit reports with runbld...'
[2020-11-11T21:05:59.951Z] Processing JUnit reports with runbld...
[2020-11-11T21:06:00.210Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-11-11T21:06:00.210Z] runbld>>> DURATION: 55ms
[2020-11-11T21:06:00.210Z] runbld>>> STDOUT: 40 bytes
[2020-11-11T21:06:00.210Z] runbld>>> STDERR: 49 bytes
[2020-11-11T21:06:00.210Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-11-11T21:06:00.210Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats_PR-22555
[2020-11-11T21:06:01.148Z] runbld>>> Storing build metadata: 
[2020-11-11T21:06:01.148Z] runbld>>> Adding test report.
[2020-11-11T21:06:01.148Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats_PR-22555/src/github.com/elastic/beats
[2020-11-11T21:06:02.100Z] runbld>>> Found 4 test output files
[2020-11-11T21:06:02.726Z] runbld>>> Test output logs contained: Errors: 0 Failures: 0 Tests: 1679 Skipped: 256
[2020-11-11T21:06:03.023Z] runbld>>> Storing result
[2020-11-11T21:06:03.023Z] runbld>>> FAILURES: 0
[2020-11-11T21:06:03.023Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-11-11T21:06:03.023Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1597739501209/t/20201111210558-24C29AB2
[2020-11-11T21:06:03.290Z] runbld>>> Email notification disabled by environment variable.
[2020-11-11T21:06:03.290Z] runbld>>> Slack notification disabled by environment variable.
[2020-11-11T21:06:08.848Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22555/src/github.com/elastic/beats
[2020-11-11T21:06:09.610Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-22555
[2020-11-11T21:06:09.845Z] [INFO] getVaultSecret: Getting secrets
[2020-11-11T21:06:10.024Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-11-11T21:06:11.358Z] + chmod 755 generate-build-data.sh
[2020-11-11T21:06:11.358Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-22555/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-22555/runs/1 ABORTED 1218725
[2020-11-11T21:06:11.358Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-22555/runs/1/steps/?limit=10000 -o steps-info.json

andrewstucki · 2020-11-11T21:25:21Z

Fixed by #19977

elasticmachine · 2020-11-11T21:55:06Z

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	4503
Skipped	561
Total	5064

elasticmachine · 2020-11-11T21:55:09Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: [Pull request #22555 updated]
Start Time: 2020-11-11T21:02:48.560+0000
Duration: 52 min 59 sec

Test stats 🧪

Test	Results
Failed	0
Passed	4503
Skipped	561
Total	5064

Disable harvester detailed metrics by default

6794fae

andrewstucki added Filebeat Filebeat breaking change labels Nov 11, 2020

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Nov 11, 2020

Update changelog

82476dd

andrewstucki added Team:Security-External Integrations Team:Services (Deprecated) Label for the former Integrations-Services team labels Nov 11, 2020

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Nov 11, 2020

andrewstucki changed the title ~~[Metricbeat] Disable harvester detailed metrics by default~~ [Filebeat] Disable harvester detailed metrics by default Nov 11, 2020

andrewstucki requested a review from kvch November 11, 2020 20:52

Update x-pack filebeat config reference

8a89b74

andrewstucki closed this Nov 11, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Filebeat] Disable harvester detailed metrics by default #22555

[Filebeat] Disable harvester detailed metrics by default #22555

andrewstucki commented Nov 11, 2020

elasticmachine commented Nov 11, 2020

elasticmachine commented Nov 11, 2020

elasticmachine commented Nov 11, 2020

Build stats

Test stats 🧪

`Error signal`

`Error signal`

`x-pack/filebeat-Lint - make -C x-pack/filebeat check;

`Error signal`

`Error signal`

`Error signal`

andrewstucki commented Nov 11, 2020

elasticmachine commented Nov 11, 2020

Test stats 🧪

elasticmachine commented Nov 11, 2020

Build stats

Test stats 🧪

[Filebeat] Disable harvester detailed metrics by default #22555

[Filebeat] Disable harvester detailed metrics by default #22555

Conversation

andrewstucki commented Nov 11, 2020

What does this PR do?

Checklist

Related issues

elasticmachine commented Nov 11, 2020

elasticmachine commented Nov 11, 2020

elasticmachine commented Nov 11, 2020

❕ Build Aborted

Build stats

Test stats 🧪

Steps errors

Error signal

Error signal

`x-pack/filebeat-Lint - make -C x-pack/filebeat check;

Error signal

Error signal

Error signal

Log output

andrewstucki commented Nov 11, 2020

elasticmachine commented Nov 11, 2020

💚 Flaky test report

Test stats 🧪

elasticmachine commented Nov 11, 2020

💚 Build Succeeded

Build stats

Test stats 🧪

`Error signal`

`Error signal`

`Error signal`

`Error signal`

`Error signal`