Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Get stats for all indices, including hidden indices #18703

Merged
merged 7 commits into from
Jul 6, 2020

Conversation

ycombinator
Copy link
Contributor

@ycombinator ycombinator commented May 22, 2020

What does this PR do?

  • Enhances the elasticsearch/index metricset to collect metrics for all indices, including hidden ones.
  • Updates all stack products' default versions used in integration tests to the latest available release.

Why is it important?

With the introduction of hidden indices in Elasticsearch (elastic/elasticsearch#50452), the GET _stats API call, by default, does not return stats for hidden indices. So it needs to be enhanced to request stats for all indices.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label May 22, 2020
@elasticmachine
Copy link
Collaborator

elasticmachine commented May 22, 2020

💔 Tests Failed

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #18703 updated]

  • Start Time: 2020-05-22T13:53:11.829+0000

  • Duration: 63 min 41 sec

Test stats 🧪

Test Results
Failed 1
Passed 3942
Skipped 934
Total 4877

Test errors

Expand to view the tests failures

  • Name: Build and Test / Metricbeat x-pack / Metricbeat x-pack / TestFetch – stats

    • Age: 2
    • Duration: 0
    • Error Details: Failed

Steps errors

Expand to view the steps failures

  • Name: Install docker-compose 1.21.0

    • Description: .ci/scripts/install-docker-compose.sh

    • Duration: 1 min 57 sec

    • Start Time: 2020-05-22T14:17:10.693+0000

    • log

  • Name: Mage build test

    • Description: mage build test

    • Duration: 35 min 2 sec

    • Start Time: 2020-05-22T14:19:43.759+0000

    • log

Log output

Expand to view the last 100 lines of log output

[2020-05-22T14:55:23.441Z] + mkdir -p /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18703/bin
[2020-05-22T14:55:23.441Z] + curl -sSLo - https://releases.hashicorp.com/terraform/0.12.24/terraform_0.12.24_linux_amd64.zip
[2020-05-22T14:55:23.707Z] ++ dirname /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18703/bin/terraform
[2020-05-22T14:55:23.707Z] + unzip -o /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18703/bin/terraform.zip -d /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18703/bin
[2020-05-22T14:55:23.707Z] Archive:  /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18703/bin/terraform.zip
[2020-05-22T14:55:24.284Z]   inflating: /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18703/bin/terraform  
[2020-05-22T14:55:24.284Z] + rm /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18703/bin/terraform.zip
[2020-05-22T14:55:24.284Z] + chmod +x /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18703/bin/terraform
[2020-05-22T14:55:24.642Z] + make mage
[2020-05-22T14:55:24.642Z] Installing mage v1.9.0 from vendor dir.
[2020-05-22T14:55:25.594Z] /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18703/.magefile cleaned
[2020-05-22T14:55:25.972Z] + git config --get user.email
[2020-05-22T14:55:25.972Z] + [ -z  ]
[2020-05-22T14:55:25.972Z] + git config user.email [email protected]
[2020-05-22T14:55:25.972Z] + git config user.name beatsmachine
[2020-05-22T14:55:26.412Z] + .ci/scripts/terraform-cleanup.sh x-pack/metricbeat
[2020-05-22T14:55:26.412Z] + DIRECTORY=x-pack/metricbeat
[2020-05-22T14:55:26.412Z] + FAILED=0
[2020-05-22T14:55:26.412Z] ++ find x-pack/metricbeat -name terraform.tfstate
[2020-05-22T14:55:26.412Z] + exit 0
[2020-05-22T14:55:26.789Z] + curl -sSLo codecov https://codecov.io/bash
[2020-05-22T14:55:27.052Z] + FILE=auditbeat/build/coverage/full.cov
[2020-05-22T14:55:27.053Z] + [ -f auditbeat/build/coverage/full.cov ]
[2020-05-22T14:55:27.053Z] + FILE=filebeat/build/coverage/full.cov
[2020-05-22T14:55:27.053Z] + [ -f filebeat/build/coverage/full.cov ]
[2020-05-22T14:55:27.053Z] + FILE=heartbeat/build/coverage/full.cov
[2020-05-22T14:55:27.053Z] + [ -f heartbeat/build/coverage/full.cov ]
[2020-05-22T14:55:27.053Z] + FILE=libbeat/build/coverage/full.cov
[2020-05-22T14:55:27.053Z] + [ -f libbeat/build/coverage/full.cov ]
[2020-05-22T14:55:27.053Z] + FILE=metricbeat/build/coverage/full.cov
[2020-05-22T14:55:27.053Z] + [ -f metricbeat/build/coverage/full.cov ]
[2020-05-22T14:55:27.053Z] + FILE=packetbeat/build/coverage/full.cov
[2020-05-22T14:55:27.053Z] + [ -f packetbeat/build/coverage/full.cov ]
[2020-05-22T14:55:27.053Z] + FILE=winlogbeat/build/coverage/full.cov
[2020-05-22T14:55:27.053Z] + [ -f winlogbeat/build/coverage/full.cov ]
[2020-05-22T14:55:27.053Z] + FILE=journalbeat/build/coverage/full.cov
[2020-05-22T14:55:27.053Z] + [ -f journalbeat/build/coverage/full.cov ]
[2020-05-22T14:55:27.446Z] Failed in branch Metricbeat x-pack
[2020-05-22T14:55:27.596Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18703/src/github.com/elastic/beats
[2020-05-22T14:55:27.915Z] + find . -type f -name TEST*.xml -path */build/* -delete
[2020-05-22T14:55:27.930Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18703/src/github.com/elastic/beats/Lint
[2020-05-22T14:55:28.017Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18703/src/github.com/elastic/beats/Metricbeat-x-pack-Mac-OS-X
[2020-05-22T14:55:28.096Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18703/src/github.com/elastic/beats/Metricbeat-OSS-Unit-tests
[2020-05-22T14:55:28.176Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18703/src/github.com/elastic/beats/Metricbeat-crosscompile
[2020-05-22T14:55:28.252Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18703/src/github.com/elastic/beats/Metricbeat-Mac-OS-X
[2020-05-22T14:55:28.327Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18703/src/github.com/elastic/beats/Metricbeat-Windows
[2020-05-22T14:55:28.427Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18703/src/github.com/elastic/beats/Filebeat-x-pack-Mac-OS-X
[2020-05-22T14:55:28.509Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18703/src/github.com/elastic/beats/Metricbeat-x-pack-Windows
[2020-05-22T14:55:28.594Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18703/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests
[2020-05-22T14:55:28.682Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18703/src/github.com/elastic/beats/Metricbeat-Python-integration-tests
[2020-05-22T14:55:28.773Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18703/src/github.com/elastic/beats/Metricbeat-x-pack
[2020-05-22T14:55:29.146Z] + cat
[2020-05-22T14:55:29.146Z] + /usr/local/bin/runbld ./runbld-script
[2020-05-22T14:55:29.146Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-05-22T14:55:37.340Z] runbld>>> runbld started
[2020-05-22T14:55:37.340Z] runbld>>> 1.6.11/a66728ff8f4356963772e6e6d2069392fa06acbe
[2020-05-22T14:55:38.295Z] runbld>>> The following profiles matched the job 'Beats/beats-beats-mbp/PR-18703' in order of occurrence in the config (last value wins).
[2020-05-22T14:55:39.691Z] runbld>>> Debug logging enabled.
[2020-05-22T14:55:39.691Z] runbld>>> Storing result
[2020-05-22T14:55:39.959Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-05-22T14:55:39.959Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200522145539-EDF8B2D4
[2020-05-22T14:55:39.959Z] runbld>>> Adding system facts.
[2020-05-22T14:55:40.921Z] runbld>>> Adding vcs info for the latest commit:  378545cf9e778f3ab5a69a939a0d1032e46d24a6
[2020-05-22T14:55:40.921Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-05-22T14:55:40.921Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-05-22T14:55:40.921Z] + echo 'Processing JUnit reports with runbld...'
[2020-05-22T14:55:40.921Z] Processing JUnit reports with runbld...
[2020-05-22T14:55:41.497Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-05-22T14:55:41.497Z] runbld>>> DURATION: 10ms
[2020-05-22T14:55:41.497Z] runbld>>> STDOUT: 40 bytes
[2020-05-22T14:55:41.497Z] runbld>>> STDERR: 49 bytes
[2020-05-22T14:55:41.497Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-05-22T14:55:41.497Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18703/src/github.com/elastic/beats
[2020-05-22T14:55:42.893Z] runbld>>> Storing build metadata: 
[2020-05-22T14:55:42.893Z] runbld>>> Adding test report.
[2020-05-22T14:55:42.893Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18703/src/github.com/elastic/beats
[2020-05-22T14:55:43.853Z] runbld>>> Found 74 test output files
[2020-05-22T14:55:44.123Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18703/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-openmetrics.xml
[2020-05-22T14:55:44.123Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18703/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-istio.xml
[2020-05-22T14:55:44.123Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18703/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-iis.xml
[2020-05-22T14:55:44.392Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18703/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-activemq.xml
[2020-05-22T14:55:44.392Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18703/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-tomcat.xml
[2020-05-22T14:55:44.392Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18703/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-cloudfoundry.xml
[2020-05-22T14:55:44.978Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18703/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests/metricbeat/build/TEST-go-integration-windows.xml
[2020-05-22T14:55:44.978Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18703/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests/metricbeat/build/TEST-go-integration-graphite.xml
[2020-05-22T14:55:45.243Z] runbld>>> Test output logs contained: Errors: 0 Failures: 1 Tests: 4877 Skipped: 729
[2020-05-22T14:55:45.509Z] runbld>>> Storing result
[2020-05-22T14:55:45.509Z] runbld>>> FAILURES: 1
[2020-05-22T14:55:45.775Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-05-22T14:55:45.775Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200522145539-EDF8B2D4
[2020-05-22T14:55:45.775Z] runbld>>> Email notification disabled by environment variable.
[2020-05-22T14:55:45.775Z] runbld>>> Slack notification disabled by environment variable.
[2020-05-22T14:55:52.204Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18703
[2020-05-22T14:55:52.528Z] [INFO] getVaultSecret: Getting secrets
[2020-05-22T14:55:52.583Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-05-22T14:55:53.293Z] + chmod 755 generate-build-data.sh
[2020-05-22T14:55:53.294Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18703/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18703/runs/5 FAILURE 3761199
[2020-05-22T14:55:53.544Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18703/runs/5/steps/?limit=10000 -o steps-info.json
[2020-05-22T14:55:54.456Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18703/runs/5/tests/?status=FAILED -o tests-errors.json
[2020-05-22T14:55:55.006Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18703/runs/5/log/ -o pipeline-log.txt

@ycombinator ycombinator changed the title Get stats for hidden indices Get stats for all indices, including hidden indices May 22, 2020
@ycombinator ycombinator added in progress Pull request is currently in progress. review Feature:Stack Monitoring labels May 22, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/stack-monitoring (Stack monitoring)

@ycombinator ycombinator added the Team:Services (Deprecated) Label for the former Integrations-Services team label May 22, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations-services (Team:Services)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label May 22, 2020
@ycombinator ycombinator force-pushed the mb-es-xp-index-hidden branch from 378545c to 3b52f14 Compare June 24, 2020 22:43
@ycombinator ycombinator marked this pull request as ready for review June 24, 2020 22:43
@ycombinator ycombinator requested a review from a team June 24, 2020 22:44
@elasticmachine
Copy link
Collaborator

elasticmachine commented Jun 24, 2020

💔 Tests Failed

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #18703 updated]

  • Start Time: 2020-07-06T16:01:24.376+0000

  • Duration: 65 min 13 sec

Test stats 🧪

Test Results
Failed 1
Passed 3488
Skipped 807
Total 4296

Test errors

Expand to view the tests failures

  • Name: Build and Test / Metricbeat x-pack / Metricbeat x-pack / TestFetch – stats

    • Age: 1
    • Duration: 0
    • Error Details: Failed

Steps errors

Expand to view the steps failures

  • Name: Mage build test
    • Description: mage build test

    • Duration: 35 min 11 sec

    • Start Time: 2020-07-06T16:27:02.131+0000

    • log

Log output

Expand to view the last 100 lines of log output

[2020-07-06T17:05:06.961Z] SUMMARY:
[2020-07-06T17:05:06.961Z]   Fail:     0
[2020-07-06T17:05:06.961Z]   Skip:     0
[2020-07-06T17:05:06.961Z]   Pass:     0
[2020-07-06T17:05:06.961Z]   Packages: 0
[2020-07-06T17:05:06.961Z]   Duration: 2.558991654s
[2020-07-06T17:05:06.961Z]   Coverage Report: /go/src/github.com/elastic/beats/metricbeat/build/TEST-go-integration-windows.html
[2020-07-06T17:05:06.961Z]   JUnit Report:    /go/src/github.com/elastic/beats/metricbeat/build/TEST-go-integration-windows.xml
[2020-07-06T17:05:06.961Z]   Output File:     /go/src/github.com/elastic/beats/metricbeat/build/TEST-go-integration-windows.out
[2020-07-06T17:05:06.961Z] >> go test: Integration-windows Test Passed
[2020-07-06T17:05:13.593Z] >> go test: Integration-zookeeper Testing
[2020-07-06T17:06:00.429Z] SUMMARY:
[2020-07-06T17:06:00.429Z]   Fail:     0
[2020-07-06T17:06:00.429Z]   Skip:     3
[2020-07-06T17:06:00.429Z]   Pass:     4
[2020-07-06T17:06:00.429Z]   Packages: 3
[2020-07-06T17:06:00.429Z]   Duration: 45.518173535s
[2020-07-06T17:06:00.429Z]   Coverage Report: /go/src/github.com/elastic/beats/metricbeat/build/TEST-go-integration-zookeeper.html
[2020-07-06T17:06:00.429Z]   JUnit Report:    /go/src/github.com/elastic/beats/metricbeat/build/TEST-go-integration-zookeeper.xml
[2020-07-06T17:06:00.429Z]   Output File:     /go/src/github.com/elastic/beats/metricbeat/build/TEST-go-integration-zookeeper.out
[2020-07-06T17:06:00.429Z] >> go test: Integration-zookeeper Test Passed
[2020-07-06T17:06:05.847Z] Recording test results
[2020-07-06T17:06:11.115Z] Stashed 39 file(s)
[2020-07-06T17:06:11.124Z] Archiving artifacts
[2020-07-06T17:06:12.401Z] + curl -sSLo codecov https://codecov.io/bash
[2020-07-06T17:06:12.699Z] + FILE=auditbeat/build/coverage/full.cov
[2020-07-06T17:06:12.699Z] + [ -f auditbeat/build/coverage/full.cov ]
[2020-07-06T17:06:12.699Z] + FILE=filebeat/build/coverage/full.cov
[2020-07-06T17:06:12.699Z] + [ -f filebeat/build/coverage/full.cov ]
[2020-07-06T17:06:12.699Z] + FILE=heartbeat/build/coverage/full.cov
[2020-07-06T17:06:12.699Z] + [ -f heartbeat/build/coverage/full.cov ]
[2020-07-06T17:06:12.699Z] + FILE=libbeat/build/coverage/full.cov
[2020-07-06T17:06:12.699Z] + [ -f libbeat/build/coverage/full.cov ]
[2020-07-06T17:06:12.699Z] + FILE=metricbeat/build/coverage/full.cov
[2020-07-06T17:06:12.699Z] + [ -f metricbeat/build/coverage/full.cov ]
[2020-07-06T17:06:12.699Z] + FILE=packetbeat/build/coverage/full.cov
[2020-07-06T17:06:12.699Z] + [ -f packetbeat/build/coverage/full.cov ]
[2020-07-06T17:06:12.699Z] + FILE=winlogbeat/build/coverage/full.cov
[2020-07-06T17:06:12.699Z] + [ -f winlogbeat/build/coverage/full.cov ]
[2020-07-06T17:06:12.699Z] + FILE=journalbeat/build/coverage/full.cov
[2020-07-06T17:06:12.699Z] + [ -f journalbeat/build/coverage/full.cov ]
[2020-07-06T17:06:14.129Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18703/src/github.com/elastic/beats
[2020-07-06T17:06:14.445Z] + find . -type f -name TEST*.xml -path */build/* -delete
[2020-07-06T17:06:14.458Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18703/src/github.com/elastic/beats/Lint
[2020-07-06T17:06:14.546Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18703/src/github.com/elastic/beats/Metricbeat-x-pack-Mac-OS-X
[2020-07-06T17:06:14.629Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18703/src/github.com/elastic/beats/Metricbeat-OSS-Unit-tests
[2020-07-06T17:06:14.715Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18703/src/github.com/elastic/beats/Metricbeat-crosscompile
[2020-07-06T17:06:14.795Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18703/src/github.com/elastic/beats/Metricbeat-Mac-OS-X
[2020-07-06T17:06:14.870Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18703/src/github.com/elastic/beats/Metricbeat-Windows
[2020-07-06T17:06:14.950Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18703/src/github.com/elastic/beats/Metricbeat-x-pack-Windows
[2020-07-06T17:06:15.031Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18703/src/github.com/elastic/beats/Metricbeat-x-pack
[2020-07-06T17:06:15.116Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18703/src/github.com/elastic/beats/Metricbeat-Python-integration-tests
[2020-07-06T17:06:15.192Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18703/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests
[2020-07-06T17:06:15.564Z] + cat
[2020-07-06T17:06:15.564Z] + /usr/local/bin/runbld ./runbld-script
[2020-07-06T17:06:15.564Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-07-06T17:06:22.150Z] runbld>>> runbld started
[2020-07-06T17:06:22.150Z] runbld>>> 1.6.12/f45d832f2ba0aa2722ab4ec1fda8ad140f027f8b
[2020-07-06T17:06:24.698Z] runbld>>> The following profiles matched the job 'Beats/beats-beats-mbp/PR-18703' in order of occurrence in the config (last value wins).
[2020-07-06T17:06:25.642Z] runbld>>> Debug logging enabled.
[2020-07-06T17:06:25.642Z] runbld>>> Storing result
[2020-07-06T17:06:25.904Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-07-06T17:06:25.904Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200706170625-091BABB7
[2020-07-06T17:06:25.904Z] runbld>>> Adding system facts.
[2020-07-06T17:06:26.869Z] runbld>>> Adding vcs info for the latest commit:  81eab48c5ed51ef48eb9066761940f519a496276
[2020-07-06T17:06:26.869Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-07-06T17:06:26.869Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-07-06T17:06:26.869Z] Processing JUnit reports with runbld...
[2020-07-06T17:06:26.869Z] + echo 'Processing JUnit reports with runbld...'
[2020-07-06T17:06:27.131Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-07-06T17:06:27.131Z] runbld>>> DURATION: 16ms
[2020-07-06T17:06:27.131Z] runbld>>> STDOUT: 40 bytes
[2020-07-06T17:06:27.131Z] runbld>>> STDERR: 49 bytes
[2020-07-06T17:06:27.131Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-07-06T17:06:27.131Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18703/src/github.com/elastic/beats
[2020-07-06T17:06:28.519Z] runbld>>> Storing build metadata: 
[2020-07-06T17:06:28.519Z] runbld>>> Adding test report.
[2020-07-06T17:06:28.519Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18703/src/github.com/elastic/beats
[2020-07-06T17:06:29.093Z] runbld>>> Found 72 test output files
[2020-07-06T17:06:29.355Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18703/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-istio.xml
[2020-07-06T17:06:29.355Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18703/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-tomcat.xml
[2020-07-06T17:06:29.355Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18703/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-iis.xml
[2020-07-06T17:06:29.355Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18703/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-openmetrics.xml
[2020-07-06T17:06:29.355Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18703/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-activemq.xml
[2020-07-06T17:06:30.299Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18703/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests/metricbeat/build/TEST-go-integration-graphite.xml
[2020-07-06T17:06:30.299Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18703/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests/metricbeat/build/TEST-go-integration-windows.xml
[2020-07-06T17:06:30.561Z] runbld>>> Test output logs contained: Errors: 0 Failures: 1 Tests: 4296 Skipped: 609
[2020-07-06T17:06:30.561Z] runbld>>> Storing result
[2020-07-06T17:06:30.561Z] runbld>>> FAILURES: 1
[2020-07-06T17:06:31.133Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-07-06T17:06:31.133Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200706170625-091BABB7
[2020-07-06T17:06:31.133Z] runbld>>> Email notification disabled by environment variable.
[2020-07-06T17:06:31.133Z] runbld>>> Slack notification disabled by environment variable.
[2020-07-06T17:06:36.650Z] Running on worker-395930 in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18703
[2020-07-06T17:06:36.776Z] [INFO] getVaultSecret: Getting secrets
[2020-07-06T17:06:36.839Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-07-06T17:06:38.857Z] + chmod 755 generate-build-data.sh
[2020-07-06T17:06:38.857Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18703/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18703/runs/14 FAILURE 3913051
[2020-07-06T17:06:38.857Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18703/runs/14/steps/?limit=10000 -o steps-info.json
[2020-07-06T17:06:40.435Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18703/runs/14/tests/?status=FAILED -o tests-errors.json

@ycombinator ycombinator force-pushed the mb-es-xp-index-hidden branch from 3b52f14 to 4e334a5 Compare July 1, 2020 07:44
Copy link
Member

@jsoriano jsoriano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, added a question about getting stats now for closed indices too, I guess we are ok with that, but just in case. Maybe we need to mention closed indices also in the changelog.

metricbeat/module/elasticsearch/index/index.go Outdated Show resolved Hide resolved
@ycombinator ycombinator force-pushed the mb-es-xp-index-hidden branch from 459e86a to 81eab48 Compare July 6, 2020 16:01
@ycombinator
Copy link
Contributor Author

@jsoriano I've addressed your review feedback regarding closed indices. This PR is ready for re-review, when you get a chance. Thanks!

@ycombinator
Copy link
Contributor Author

CI failures are unrelated. Merging.

@ycombinator ycombinator merged commit e88743f into elastic:master Jul 6, 2020
@ycombinator ycombinator deleted the mb-es-xp-index-hidden branch July 6, 2020 17:20
@ycombinator ycombinator added v7.9.0 v8.0.0 and removed in progress Pull request is currently in progress. labels Jul 8, 2020
ycombinator added a commit that referenced this pull request Jul 8, 2020
* Get stats for all indices

* Update default stack versions to use in integration tests

* Add integration test for hidden indices stats

* Better naming of variables

* Adding CHANGELOG entry

* Use expand_wildcards in tests as well

* Collect stats for open and hidden indices (but not closed ones)
melchiormoulin pushed a commit to melchiormoulin/beats that referenced this pull request Oct 14, 2020
* Get stats for all indices

* Update default stack versions to use in integration tests

* Add integration test for hidden indices stats

* Better naming of variables

* Adding CHANGELOG entry

* Use expand_wildcards in tests as well

* Collect stats for open and hidden indices (but not closed ones)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Feature:Stack Monitoring review Team:Services (Deprecated) Label for the former Integrations-Services team v7.9.0 v8.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants