Metricbeat, fix metricsets log level behavior #27803

philippkahr · 2021-09-08T09:04:33Z

What does this PR do?

Until now errors in the metric sets were only reported as INFO. If someone configures metricbeat to only write a log file on error, issues like cannot connect to Elasticsearch 404, 400 Bad request could be missed.

Why is it important?

Easier debugging
If metricbeat cannot connect it is an error and should be classified as such a thing.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
[] I have made corresponding changes to the documentation
[ ]~~ I have made corresponding change to the default configuration files~~
~~I have added tests that prove my fix is effective or that my feature works~~
[] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

Closes Metricbeat metric sets modules uses wrong log level #27799

…c#27799

botelastic · 2021-09-08T09:04:39Z

This pull request doesn't have a Team:<team> label.

This reverts commit 8dceae3.

This reverts commit 044055d.

This reverts commit 355f823.

elasticmachine · 2021-09-08T09:20:31Z

❕ Build Aborted

There is a new build on-going so the previous on-going builds have been aborted.

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Reason: Aborted from #2
Start Time: 2021-09-08T09:04:48.824+0000
Duration: 15 min 28 sec
Commit: 8173f9b

Trends 🧪

Steps errors

Expand to view the steps failures

`List files to upload`

Took 0 min 0 sec . View more details on here
Description: ls -l src/github.com/elastic/beats/build/system-tests-*.tar.gz

`List files to upload`

Took 0 min 0 sec . View more details on here
Description: ls -l src/github.com/elastic/beats/build/system-tests-*.tar.gz

`List files to upload`

Took 0 min 0 sec . View more details on here
Description: ls -l src/github.com/elastic/beats/build/system-tests-*.tar.gz

`List files to upload`

Took 0 min 0 sec . View more details on here
Description: ls -l src/github.com/elastic/beats/build/system-tests-*.tar.gz

`Error signal`

Took 0 min 0 sec . View more details on here
Description: Error 'org.jenkinsci.plugins.workflow.steps.FlowInterruptedException'

Log output

Expand to view the last 100 lines of log output

[2021-09-08T09:19:48.676Z] Running in /var/lib/jenkins/workspace/PR-27803-1-216dffe6-5467-456f-bb1b-7ac470175d0e/src/github.com/elastic/beats/src/github.com/elastic/beats
[2021-09-08T09:19:48.889Z] Running in /var/lib/jenkins/workspace/PR-27803-1-42bf98be-2e2a-4116-bc5c-224e56c2f715/src/github.com/elastic/beats/src/github.com/elastic/beats
[2021-09-08T09:19:49.062Z] + ls -l src/github.com/elastic/beats/build/system-tests-*.tar.gz
[2021-09-08T09:19:49.062Z] ls: cannot access 'src/github.com/elastic/beats/build/system-tests-*.tar.gz': No such file or directory
[2021-09-08T09:19:49.163Z] ERROR: Failed to archive the build test results
[2021-09-08T09:19:49.163Z] ERROR: script returned exit code 2
[2021-09-08T09:19:49.214Z] Timeout set to expire in 5 min 0 sec
[2021-09-08T09:19:49.243Z] + ls -l src/github.com/elastic/beats/build/system-tests-*.tar.gz
[2021-09-08T09:19:49.243Z] ls: cannot access 'src/github.com/elastic/beats/build/system-tests-*.tar.gz': No such file or directory
[2021-09-08T09:19:49.552Z] ERROR: Failed to archive the build test results
[2021-09-08T09:19:49.552Z] ERROR: script returned exit code 2
[2021-09-08T09:19:49.591Z] Cleaning up /var/lib/jenkins/workspace/PR-27803-1-04563421-9402-41f9-97da-bdbd2dbf1d8b
[2021-09-08T09:19:49.591Z] Change ownership of all files inside the specific folder from root/root to current user/group
[2021-09-08T09:19:49.591Z] ++ id -u
[2021-09-08T09:19:49.591Z] ++ id -g
[2021-09-08T09:19:49.591Z] + docker run -v /var/lib/jenkins/workspace/PR-27803-1-04563421-9402-41f9-97da-bdbd2dbf1d8b:/beat alpine:3.4 sh -c 'find /beat -user 0 -exec chown -h 1171:1172 {} \;'
[2021-09-08T09:19:49.890Z] + go clean -modcache
[2021-09-08T09:19:50.128Z] + go clean -modcache
[2021-09-08T09:19:50.534Z] + set -e
[2021-09-08T09:19:50.534Z] + echo 'Change permissions with write access of all files inside the specific folder'
[2021-09-08T09:19:50.534Z] Change permissions with write access of all files inside the specific folder
[2021-09-08T09:19:50.535Z] + chmod -R +w /var/lib/jenkins/workspace/PR-27803-1-04563421-9402-41f9-97da-bdbd2dbf1d8b
[2021-09-08T09:19:50.910Z] Running in /var/lib/jenkins/workspace/PR-27803-1-04563421-9402-41f9-97da-bdbd2dbf1d8b
[2021-09-08T09:19:53.256Z] Timeout set to expire in 5 min 0 sec
[2021-09-08T09:19:53.481Z] Timeout set to expire in 5 min 0 sec
[2021-09-08T09:19:53.608Z] Cleaning up /var/lib/jenkins/workspace/PR-27803-1-216dffe6-5467-456f-bb1b-7ac470175d0e
[2021-09-08T09:19:53.608Z] Change ownership of all files inside the specific folder from root/root to current user/group
[2021-09-08T09:19:53.608Z] ++ id -u
[2021-09-08T09:19:53.608Z] ++ id -g
[2021-09-08T09:19:53.608Z] + docker run -v /var/lib/jenkins/workspace/PR-27803-1-216dffe6-5467-456f-bb1b-7ac470175d0e:/beat alpine:3.4 sh -c 'find /beat -user 0 -exec chown -h 1171:1172 {} \;'
[2021-09-08T09:19:53.827Z] Cleaning up /var/lib/jenkins/workspace/PR-27803-1-42bf98be-2e2a-4116-bc5c-224e56c2f715
[2021-09-08T09:19:53.827Z] Change ownership of all files inside the specific folder from root/root to current user/group
[2021-09-08T09:19:53.827Z] ++ id -u
[2021-09-08T09:19:53.827Z] ++ id -g
[2021-09-08T09:19:53.827Z] + docker run -v /var/lib/jenkins/workspace/PR-27803-1-42bf98be-2e2a-4116-bc5c-224e56c2f715:/beat alpine:3.4 sh -c 'find /beat -user 0 -exec chown -h 1171:1172 {} \;'
[2021-09-08T09:19:54.084Z] Failed in branch auditbeat-lint
[2021-09-08T09:19:54.556Z] + set -e
[2021-09-08T09:19:54.557Z] + echo 'Change permissions with write access of all files inside the specific folder'
[2021-09-08T09:19:54.557Z] Change permissions with write access of all files inside the specific folder
[2021-09-08T09:19:54.557Z] + chmod -R +w /var/lib/jenkins/workspace/PR-27803-1-216dffe6-5467-456f-bb1b-7ac470175d0e
[2021-09-08T09:19:54.768Z] + set -e
[2021-09-08T09:19:54.768Z] + echo 'Change permissions with write access of all files inside the specific folder'
[2021-09-08T09:19:54.768Z] Change permissions with write access of all files inside the specific folder
[2021-09-08T09:19:54.768Z] + chmod -R +w /var/lib/jenkins/workspace/PR-27803-1-42bf98be-2e2a-4116-bc5c-224e56c2f715
[2021-09-08T09:19:54.925Z] >> there are no system test files under build/system-tests/run
[2021-09-08T09:19:54.935Z] Running in /var/lib/jenkins/workspace/PR-27803-1-216dffe6-5467-456f-bb1b-7ac470175d0e
[2021-09-08T09:19:55.166Z] Running in /var/lib/jenkins/workspace/PR-27803-1-95b1741b-c8df-42c4-843c-25b7f49d7a51/src/github.com/elastic/beats/src/github.com/elastic/beats
[2021-09-08T09:19:55.263Z] Running in /var/lib/jenkins/workspace/PR-27803-1-42bf98be-2e2a-4116-bc5c-224e56c2f715
[2021-09-08T09:19:55.666Z] + ls -l src/github.com/elastic/beats/build/system-tests-*.tar.gz
[2021-09-08T09:19:55.666Z] ls: cannot access 'src/github.com/elastic/beats/build/system-tests-*.tar.gz': No such file or directory
[2021-09-08T09:19:55.762Z] ERROR: Failed to archive the build test results
[2021-09-08T09:19:55.762Z] ERROR: script returned exit code 2
[2021-09-08T09:19:56.381Z] + go clean -modcache
[2021-09-08T09:19:58.158Z] Failed in branch x-pack/auditbeat-lint
[2021-09-08T09:19:58.728Z] Failed in branch x-pack/metricbeat-lint
[2021-09-08T09:19:59.770Z] Timeout set to expire in 5 min 0 sec
[2021-09-08T09:20:00.139Z] Cleaning up /var/lib/jenkins/workspace/PR-27803-1-95b1741b-c8df-42c4-843c-25b7f49d7a51
[2021-09-08T09:20:00.139Z] Change ownership of all files inside the specific folder from root/root to current user/group
[2021-09-08T09:20:00.139Z] ++ id -u
[2021-09-08T09:20:00.139Z] ++ id -g
[2021-09-08T09:20:00.139Z] + docker run -v /var/lib/jenkins/workspace/PR-27803-1-95b1741b-c8df-42c4-843c-25b7f49d7a51:/beat alpine:3.4 sh -c 'find /beat -user 0 -exec chown -h 1171:1172 {} \;'
[2021-09-08T09:20:01.084Z] + set -e
[2021-09-08T09:20:01.084Z] + echo 'Change permissions with write access of all files inside the specific folder'
[2021-09-08T09:20:01.084Z] Change permissions with write access of all files inside the specific folder
[2021-09-08T09:20:01.084Z] + chmod -R +w /var/lib/jenkins/workspace/PR-27803-1-95b1741b-c8df-42c4-843c-25b7f49d7a51
[2021-09-08T09:20:01.255Z] Running in /var/lib/jenkins/workspace/PR-27803-1-95b1741b-c8df-42c4-843c-25b7f49d7a51
[2021-09-08T09:20:05.073Z] Failed in branch metricbeat-lint
[2021-09-08T09:20:05.707Z] + go clean -modcache
[2021-09-08T09:20:09.062Z] Timeout set to expire in 5 min 0 sec
[2021-09-08T09:20:09.400Z] Cleaning up /var/lib/jenkins/workspace/Beats_beats_PR-27803
[2021-09-08T09:20:09.400Z] Change ownership of all files inside the specific folder from root/root to current user/group
[2021-09-08T09:20:09.400Z] ++ id -u
[2021-09-08T09:20:09.400Z] ++ id -g
[2021-09-08T09:20:09.400Z] + docker run -v /var/lib/jenkins/workspace/Beats_beats_PR-27803:/beat alpine:3.4 sh -c 'find /beat -user 0 -exec chown -h 1171:1172 {} \;'
[2021-09-08T09:20:09.400Z] Unable to find image 'alpine:3.4' locally
[2021-09-08T09:20:09.970Z] 3.4: Pulling from library/alpine
[2021-09-08T09:20:10.230Z] c1e54eec4b57: Pulling fs layer
[2021-09-08T09:20:10.492Z] c1e54eec4b57: Download complete
[2021-09-08T09:20:10.753Z] c1e54eec4b57: Pull complete
[2021-09-08T09:20:10.753Z] Digest: sha256:b733d4a32c4da6a00a84df2ca32791bb03df95400243648d8c539e7b4cce329c
[2021-09-08T09:20:10.753Z] Status: Downloaded newer image for alpine:3.4
[2021-09-08T09:20:11.696Z] + set -e
[2021-09-08T09:20:11.696Z] + echo 'Change permissions with write access of all files inside the specific folder'
[2021-09-08T09:20:11.696Z] Change permissions with write access of all files inside the specific folder
[2021-09-08T09:20:11.696Z] + chmod -R +w /var/lib/jenkins/workspace/Beats_beats_PR-27803
[2021-09-08T09:20:11.797Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-27803
[2021-09-08T09:20:15.235Z] Stage "Build&Test" skipped due to earlier failure(s)
[2021-09-08T09:20:15.345Z] Stage "Extended" skipped due to earlier failure(s)
[2021-09-08T09:20:15.453Z] Stage "Packaging" skipped due to earlier failure(s)
[2021-09-08T09:20:15.564Z] Stage "Packaging-Pipeline" skipped due to earlier failure(s)
[2021-09-08T09:20:15.708Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-27803/src/github.com/elastic/beats
[2021-09-08T09:20:16.088Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-27803
[2021-09-08T09:20:16.206Z] [INFO] getVaultSecret: Getting secrets
[2021-09-08T09:20:16.265Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-09-08T09:20:17.343Z] + chmod 755 generate-build-data.sh
[2021-09-08T09:20:17.343Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-27803/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-27803/runs/1 ABORTED 928240
[2021-09-08T09:20:17.343Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-27803/runs/1/steps/?limit=10000 -o steps-info.json
[2021-09-08T09:20:18.254Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-27803/runs/1/tests/?status=FAILED -o tests-errors.json
[2021-09-08T09:20:18.254Z] Retry 1/3 exited 22, retrying in 1 seconds...
[2021-09-08T09:20:19.165Z] Retry 2/3 exited 22, retrying in 2 seconds...

philippkahr and others added 6 commits March 22, 2021 10:50

fixed error from beats about non iterable list

8dceae3

run make update elastic#24665

044055d

fixed CI issue forcing me todo a make update

355f823

Merge remote-tracking branch 'upstream/master'

24036b8

Merge branch 'elastic:master' into master

6c197f9

MetricSets should throw an ERROR for log level instead of INFO elasti…

8173f9b

…c#27799

philippkahr added enhancement cleanup labels Sep 8, 2021

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Sep 8, 2021

philippkahr added 3 commits September 8, 2021 11:12

Revert "fixed error from beats about non iterable list"

73fc6c9

This reverts commit 8dceae3.

Revert "run make update elastic#24665"

3146b19

This reverts commit 044055d.

Revert "fixed CI issue forcing me todo a make update"

0c42b91

This reverts commit 355f823.

philippkahr closed this Sep 8, 2021

philippkahr deleted the fix-error-level-of-metricsets branch September 8, 2021 09:17

philippkahr mentioned this pull request Sep 8, 2021

Metricbeat, fix metricsets log level behavior #27804

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Metricbeat, fix metricsets log level behavior #27803

Metricbeat, fix metricsets log level behavior #27803

philippkahr commented Sep 8, 2021

botelastic bot commented Sep 8, 2021

elasticmachine commented Sep 8, 2021

Build stats

Trends 🧪

`List files to upload`

`List files to upload`

`List files to upload`

`List files to upload`

`Error signal`

Metricbeat, fix metricsets log level behavior #27803

Metricbeat, fix metricsets log level behavior #27803

Conversation

philippkahr commented Sep 8, 2021

What does this PR do?

Why is it important?

Checklist

Related issues

botelastic bot commented Sep 8, 2021

elasticmachine commented Sep 8, 2021

❕ Build Aborted

Build stats

Trends 🧪

Steps errors

List files to upload

List files to upload

List files to upload

List files to upload

Error signal

Log output

`List files to upload`

`List files to upload`

`List files to upload`

`List files to upload`

`Error signal`