Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Report if index is hidden in index stats #18706

Merged
merged 5 commits into from
Jul 8, 2020

Conversation

ycombinator
Copy link
Contributor

@ycombinator ycombinator commented May 22, 2020

What does this PR do?

Enhances the elasticsearch/index metricset to report if an index is hidden or not.

Why is it important?

This will allow the Stack Monitoring UI to suitably mark the index in the UI.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label May 22, 2020
@elasticmachine
Copy link
Collaborator

elasticmachine commented May 22, 2020

💔 Tests Failed

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #18706 updated]

  • Start Time: 2020-05-22T13:54:54.518+0000

  • Duration: 68 min 19 sec

Test stats 🧪

Test Results
Failed 1
Passed 3942
Skipped 934
Total 4877

Test errors

Expand to view the tests failures

  • Name: Build and Test / Metricbeat x-pack / Metricbeat x-pack / TestFetch – stats

    • Age: 1
    • Duration: 0
    • Error Details: Failed

Steps errors

Expand to view the steps failures

  • Name: Mage build test
    • Description: mage build test

    • Duration: 39 min 37 sec

    • Start Time: 2020-05-22T14:23:26.464+0000

    • log

Log output

Expand to view the last 100 lines of log output

[2020-05-22T15:02:44.781Z] + OS=linux
[2020-05-22T15:02:44.781Z] + mkdir -p /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/bin
[2020-05-22T15:02:44.781Z] + curl -sSLo - https://releases.hashicorp.com/terraform/0.12.24/terraform_0.12.24_linux_amd64.zip
[2020-05-22T15:02:45.047Z] ++ dirname /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/bin/terraform
[2020-05-22T15:02:45.047Z] + unzip -o /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/bin/terraform.zip -d /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/bin
[2020-05-22T15:02:45.047Z] Archive:  /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/bin/terraform.zip
[2020-05-22T15:02:45.627Z]   inflating: /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/bin/terraform  
[2020-05-22T15:02:45.627Z] + rm /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/bin/terraform.zip
[2020-05-22T15:02:45.627Z] + chmod +x /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/bin/terraform
[2020-05-22T15:02:45.987Z] + make mage
[2020-05-22T15:02:45.987Z] Installing mage v1.9.0 from vendor dir.
[2020-05-22T15:02:46.943Z] /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/.magefile cleaned
[2020-05-22T15:02:47.285Z] + git config --get user.email
[2020-05-22T15:02:47.286Z] + [ -z  ]
[2020-05-22T15:02:47.286Z] + git config user.email [email protected]
[2020-05-22T15:02:47.286Z] + git config user.name beatsmachine
[2020-05-22T15:02:47.704Z] + .ci/scripts/terraform-cleanup.sh x-pack/metricbeat
[2020-05-22T15:02:47.705Z] + DIRECTORY=x-pack/metricbeat
[2020-05-22T15:02:47.705Z] + FAILED=0
[2020-05-22T15:02:47.705Z] ++ find x-pack/metricbeat -name terraform.tfstate
[2020-05-22T15:02:47.705Z] + exit 0
[2020-05-22T15:02:48.090Z] + curl -sSLo codecov https://codecov.io/bash
[2020-05-22T15:02:48.357Z] + FILE=auditbeat/build/coverage/full.cov
[2020-05-22T15:02:48.357Z] + [ -f auditbeat/build/coverage/full.cov ]
[2020-05-22T15:02:48.357Z] + FILE=filebeat/build/coverage/full.cov
[2020-05-22T15:02:48.357Z] + [ -f filebeat/build/coverage/full.cov ]
[2020-05-22T15:02:48.357Z] + FILE=heartbeat/build/coverage/full.cov
[2020-05-22T15:02:48.357Z] + [ -f heartbeat/build/coverage/full.cov ]
[2020-05-22T15:02:48.357Z] + FILE=libbeat/build/coverage/full.cov
[2020-05-22T15:02:48.357Z] + [ -f libbeat/build/coverage/full.cov ]
[2020-05-22T15:02:48.357Z] + FILE=metricbeat/build/coverage/full.cov
[2020-05-22T15:02:48.357Z] + [ -f metricbeat/build/coverage/full.cov ]
[2020-05-22T15:02:48.357Z] + FILE=packetbeat/build/coverage/full.cov
[2020-05-22T15:02:48.357Z] + [ -f packetbeat/build/coverage/full.cov ]
[2020-05-22T15:02:48.358Z] + FILE=winlogbeat/build/coverage/full.cov
[2020-05-22T15:02:48.358Z] + [ -f winlogbeat/build/coverage/full.cov ]
[2020-05-22T15:02:48.358Z] + FILE=journalbeat/build/coverage/full.cov
[2020-05-22T15:02:48.358Z] + [ -f journalbeat/build/coverage/full.cov ]
[2020-05-22T15:02:48.797Z] Failed in branch Metricbeat x-pack
[2020-05-22T15:02:48.953Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/src/github.com/elastic/beats
[2020-05-22T15:02:49.282Z] + find . -type f -name TEST*.xml -path */build/* -delete
[2020-05-22T15:02:49.300Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/src/github.com/elastic/beats/Lint
[2020-05-22T15:02:49.403Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/src/github.com/elastic/beats/Filebeat-x-pack-Mac-OS-X
[2020-05-22T15:02:49.486Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/src/github.com/elastic/beats/Metricbeat-x-pack-Mac-OS-X
[2020-05-22T15:02:49.573Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/src/github.com/elastic/beats/Metricbeat-Mac-OS-X
[2020-05-22T15:02:49.653Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/src/github.com/elastic/beats/Metricbeat-crosscompile
[2020-05-22T15:02:49.747Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/src/github.com/elastic/beats/Metricbeat-OSS-Unit-tests
[2020-05-22T15:02:49.838Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/src/github.com/elastic/beats/Metricbeat-Windows
[2020-05-22T15:02:49.948Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/src/github.com/elastic/beats/Metricbeat-x-pack-Windows
[2020-05-22T15:02:50.052Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests
[2020-05-22T15:02:50.140Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/src/github.com/elastic/beats/Metricbeat-Python-integration-tests
[2020-05-22T15:02:50.234Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/src/github.com/elastic/beats/Metricbeat-x-pack
[2020-05-22T15:02:50.618Z] + cat
[2020-05-22T15:02:50.618Z] + /usr/local/bin/runbld ./runbld-script
[2020-05-22T15:02:50.618Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-05-22T15:02:58.829Z] runbld>>> runbld started
[2020-05-22T15:02:58.829Z] runbld>>> 1.6.11/a66728ff8f4356963772e6e6d2069392fa06acbe
[2020-05-22T15:02:59.792Z] runbld>>> The following profiles matched the job 'Beats/beats-beats-mbp/PR-18706' in order of occurrence in the config (last value wins).
[2020-05-22T15:03:01.198Z] runbld>>> Debug logging enabled.
[2020-05-22T15:03:01.198Z] runbld>>> Storing result
[2020-05-22T15:03:01.198Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-05-22T15:03:01.465Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200522150300-6523BC25
[2020-05-22T15:03:01.465Z] runbld>>> Adding system facts.
[2020-05-22T15:03:02.430Z] runbld>>> Adding vcs info for the latest commit:  22de4c2cc32782202cc30ed8b0403add380d4a74
[2020-05-22T15:03:02.430Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-05-22T15:03:02.430Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-05-22T15:03:02.430Z] + echo 'Processing JUnit reports with runbld...'
[2020-05-22T15:03:02.430Z] Processing JUnit reports with runbld...
[2020-05-22T15:03:03.012Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-05-22T15:03:03.012Z] runbld>>> DURATION: 12ms
[2020-05-22T15:03:03.012Z] runbld>>> STDOUT: 40 bytes
[2020-05-22T15:03:03.012Z] runbld>>> STDERR: 49 bytes
[2020-05-22T15:03:03.012Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-05-22T15:03:03.012Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/src/github.com/elastic/beats
[2020-05-22T15:03:04.418Z] runbld>>> Storing build metadata: 
[2020-05-22T15:03:04.418Z] runbld>>> Adding test report.
[2020-05-22T15:03:04.418Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/src/github.com/elastic/beats
[2020-05-22T15:03:05.398Z] runbld>>> Found 74 test output files
[2020-05-22T15:03:05.688Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-openmetrics.xml
[2020-05-22T15:03:05.953Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-istio.xml
[2020-05-22T15:03:05.953Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-iis.xml
[2020-05-22T15:03:05.953Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-activemq.xml
[2020-05-22T15:03:05.953Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-tomcat.xml
[2020-05-22T15:03:05.953Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-cloudfoundry.xml
[2020-05-22T15:03:06.531Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests/metricbeat/build/TEST-go-integration-windows.xml
[2020-05-22T15:03:06.531Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests/metricbeat/build/TEST-go-integration-graphite.xml
[2020-05-22T15:03:07.112Z] runbld>>> Test output logs contained: Errors: 0 Failures: 1 Tests: 4877 Skipped: 729
[2020-05-22T15:03:07.112Z] runbld>>> Storing result
[2020-05-22T15:03:07.112Z] runbld>>> FAILURES: 1
[2020-05-22T15:03:07.691Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-05-22T15:03:07.691Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200522150300-6523BC25
[2020-05-22T15:03:07.691Z] runbld>>> Email notification disabled by environment variable.
[2020-05-22T15:03:07.691Z] runbld>>> Slack notification disabled by environment variable.
[2020-05-22T15:03:13.285Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706
[2020-05-22T15:03:13.407Z] [INFO] getVaultSecret: Getting secrets
[2020-05-22T15:03:13.464Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-05-22T15:03:14.217Z] + chmod 755 generate-build-data.sh
[2020-05-22T15:03:14.217Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18706/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18706/runs/3 FAILURE 4099435
[2020-05-22T15:03:14.768Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18706/runs/3/steps/?limit=10000 -o steps-info.json
[2020-05-22T15:03:15.319Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18706/runs/3/tests/?status=FAILED -o tests-errors.json

@ycombinator ycombinator force-pushed the mb-es-xp-index-mark-hidden branch from f487f0c to 22de4c2 Compare May 22, 2020 13:54
@ycombinator ycombinator added in progress Pull request is currently in progress. Feature:Stack Monitoring Team:Services (Deprecated) Label for the former Integrations-Services team labels May 22, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations-services (Team:Services)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label May 22, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/stack-monitoring (Stack monitoring)

@ycombinator ycombinator force-pushed the mb-es-xp-index-mark-hidden branch from 22de4c2 to c07c160 Compare July 6, 2020 17:25
@ycombinator ycombinator marked this pull request as ready for review July 6, 2020 17:26
@ycombinator ycombinator removed the in progress Pull request is currently in progress. label Jul 6, 2020
@ycombinator ycombinator requested a review from a team July 6, 2020 17:26
@elasticmachine
Copy link
Collaborator

elasticmachine commented Jul 6, 2020

💔 Tests Failed

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #18706 updated]

  • Start Time: 2020-07-07T15:43:36.730+0000

  • Duration: 64 min 30 sec

Test stats 🧪

Test Results
Failed 1
Passed 2992
Skipped 696
Total 3689

Test errors

Expand to view the tests failures

  • Name: Build and Test / Metricbeat x-pack / Metricbeat x-pack / TestFetch – stats

    • Age: 2
    • Duration: 0
    • Error Details: Failed

Steps errors

Expand to view the steps failures

  • Name: Install Go 1.13.10

    • Description: .ci/scripts/install-go.sh

    • Duration: 2 min 22 sec

    • Start Time: 2020-07-07T16:08:06.683+0000

    • log

  • Name: Mage build test

    • Description: mage build test

    • Duration: 36 min 13 sec

    • Start Time: 2020-07-07T16:10:42.173+0000

    • log

Log output

Expand to view the last 100 lines of log output

[2020-07-07T16:47:37.812Z] + OS=linux
[2020-07-07T16:47:37.812Z] + mkdir -p /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/bin
[2020-07-07T16:47:37.812Z] + curl -sSLo - https://releases.hashicorp.com/terraform/0.12.24/terraform_0.12.24_linux_amd64.zip
[2020-07-07T16:47:38.766Z] ++ dirname /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/bin/terraform
[2020-07-07T16:47:38.766Z] + unzip -o /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/bin/terraform.zip -d /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/bin
[2020-07-07T16:47:38.766Z] Archive:  /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/bin/terraform.zip
[2020-07-07T16:47:39.029Z]   inflating: /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/bin/terraform  
[2020-07-07T16:47:39.029Z] + rm /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/bin/terraform.zip
[2020-07-07T16:47:39.029Z] + chmod +x /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/bin/terraform
[2020-07-07T16:47:39.392Z] + make mage
[2020-07-07T16:47:39.392Z] Installing mage v1.9.0.
[2020-07-07T16:47:40.345Z] go: finding github.com/magefile/mage v1.9.0
[2020-07-07T16:47:40.345Z] go: downloading github.com/magefile/mage v1.9.0
[2020-07-07T16:47:40.608Z] go: extracting github.com/magefile/mage v1.9.0
[2020-07-07T16:47:42.003Z] /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/.magefile cleaned
[2020-07-07T16:47:42.345Z] + git config --get user.email
[2020-07-07T16:47:42.345Z] + [ -z  ]
[2020-07-07T16:47:42.345Z] + git config user.email [email protected]
[2020-07-07T16:47:42.345Z] + git config user.name beatsmachine
[2020-07-07T16:47:42.829Z] + .ci/scripts/terraform-cleanup.sh x-pack/metricbeat
[2020-07-07T16:47:42.829Z] + DIRECTORY=x-pack/metricbeat
[2020-07-07T16:47:42.829Z] + FAILED=0
[2020-07-07T16:47:42.829Z] ++ find x-pack/metricbeat -name terraform.tfstate
[2020-07-07T16:47:42.829Z] + exit 0
[2020-07-07T16:47:43.224Z] + curl -sSLo codecov https://codecov.io/bash
[2020-07-07T16:47:43.486Z] + FILE=auditbeat/build/coverage/full.cov
[2020-07-07T16:47:43.486Z] + [ -f auditbeat/build/coverage/full.cov ]
[2020-07-07T16:47:43.486Z] + FILE=filebeat/build/coverage/full.cov
[2020-07-07T16:47:43.486Z] + [ -f filebeat/build/coverage/full.cov ]
[2020-07-07T16:47:43.486Z] + FILE=heartbeat/build/coverage/full.cov
[2020-07-07T16:47:43.486Z] + [ -f heartbeat/build/coverage/full.cov ]
[2020-07-07T16:47:43.486Z] + FILE=libbeat/build/coverage/full.cov
[2020-07-07T16:47:43.486Z] + [ -f libbeat/build/coverage/full.cov ]
[2020-07-07T16:47:43.486Z] + FILE=metricbeat/build/coverage/full.cov
[2020-07-07T16:47:43.486Z] + [ -f metricbeat/build/coverage/full.cov ]
[2020-07-07T16:47:43.486Z] + FILE=packetbeat/build/coverage/full.cov
[2020-07-07T16:47:43.486Z] + [ -f packetbeat/build/coverage/full.cov ]
[2020-07-07T16:47:43.486Z] + FILE=winlogbeat/build/coverage/full.cov
[2020-07-07T16:47:43.486Z] + [ -f winlogbeat/build/coverage/full.cov ]
[2020-07-07T16:47:43.486Z] + FILE=journalbeat/build/coverage/full.cov
[2020-07-07T16:47:43.486Z] + [ -f journalbeat/build/coverage/full.cov ]
[2020-07-07T16:47:43.863Z] Failed in branch Metricbeat x-pack
[2020-07-07T16:47:44.021Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/src/github.com/elastic/beats
[2020-07-07T16:47:44.325Z] + find . -type f -name TEST*.xml -path */build/* -delete
[2020-07-07T16:47:44.336Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/src/github.com/elastic/beats/Lint
[2020-07-07T16:47:44.441Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests
[2020-07-07T16:47:44.521Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/src/github.com/elastic/beats/Metricbeat-Mac-OS-X
[2020-07-07T16:47:44.609Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/src/github.com/elastic/beats/Metricbeat-Python-integration-tests
[2020-07-07T16:47:44.692Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/src/github.com/elastic/beats/Metricbeat-x-pack-Mac-OS-X
[2020-07-07T16:47:44.770Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/src/github.com/elastic/beats/Metricbeat-OSS-Unit-tests
[2020-07-07T16:47:44.865Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/src/github.com/elastic/beats/Metricbeat-crosscompile
[2020-07-07T16:47:44.951Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/src/github.com/elastic/beats/Metricbeat-Windows
[2020-07-07T16:47:45.044Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/src/github.com/elastic/beats/Metricbeat-x-pack-Windows
[2020-07-07T16:47:45.131Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/src/github.com/elastic/beats/Metricbeat-x-pack
[2020-07-07T16:47:45.499Z] + cat
[2020-07-07T16:47:45.499Z] + /usr/local/bin/runbld ./runbld-script
[2020-07-07T16:47:45.499Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-07-07T16:47:52.084Z] runbld>>> runbld started
[2020-07-07T16:47:52.084Z] runbld>>> 1.6.12/f45d832f2ba0aa2722ab4ec1fda8ad140f027f8b
[2020-07-07T16:47:53.465Z] runbld>>> The following profiles matched the job 'Beats/beats-beats-mbp/PR-18706' in order of occurrence in the config (last value wins).
[2020-07-07T16:47:54.888Z] runbld>>> Debug logging enabled.
[2020-07-07T16:47:54.888Z] runbld>>> Storing result
[2020-07-07T16:47:54.888Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-07-07T16:47:54.888Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200707164754-471AD83E
[2020-07-07T16:47:54.888Z] runbld>>> Adding system facts.
[2020-07-07T16:47:55.832Z] runbld>>> Adding vcs info for the latest commit:  fed82ab67be8ef95e41c9351a96630097b6fede4
[2020-07-07T16:47:55.832Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-07-07T16:47:55.832Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-07-07T16:47:55.832Z] + echo 'Processing JUnit reports with runbld...'
[2020-07-07T16:47:55.832Z] Processing JUnit reports with runbld...
[2020-07-07T16:47:56.405Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-07-07T16:47:56.405Z] runbld>>> DURATION: 34ms
[2020-07-07T16:47:56.405Z] runbld>>> STDOUT: 40 bytes
[2020-07-07T16:47:56.405Z] runbld>>> STDERR: 49 bytes
[2020-07-07T16:47:56.405Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-07-07T16:47:56.405Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/src/github.com/elastic/beats
[2020-07-07T16:47:57.343Z] runbld>>> Storing build metadata: 
[2020-07-07T16:47:57.343Z] runbld>>> Adding test report.
[2020-07-07T16:47:57.344Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/src/github.com/elastic/beats
[2020-07-07T16:47:58.288Z] runbld>>> Found 34 test output files
[2020-07-07T16:47:58.288Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-istio.xml
[2020-07-07T16:47:58.547Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-tomcat.xml
[2020-07-07T16:47:58.547Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-iis.xml
[2020-07-07T16:47:58.547Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-openmetrics.xml
[2020-07-07T16:47:58.547Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-activemq.xml
[2020-07-07T16:47:59.115Z] runbld>>> Test output logs contained: Errors: 0 Failures: 1 Tests: 3689 Skipped: 597
[2020-07-07T16:47:59.374Z] runbld>>> Storing result
[2020-07-07T16:47:59.374Z] runbld>>> FAILURES: 1
[2020-07-07T16:47:59.633Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-07-07T16:47:59.633Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200707164754-471AD83E
[2020-07-07T16:47:59.892Z] runbld>>> Email notification disabled by environment variable.
[2020-07-07T16:47:59.892Z] runbld>>> Slack notification disabled by environment variable.
[2020-07-07T16:48:05.713Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18706
[2020-07-07T16:48:05.874Z] [INFO] getVaultSecret: Getting secrets
[2020-07-07T16:48:05.950Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-07-07T16:48:06.805Z] + chmod 755 generate-build-data.sh
[2020-07-07T16:48:06.805Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18706/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18706/runs/5 FAILURE 3869813
[2020-07-07T16:48:06.805Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18706/runs/5/steps/?limit=10000 -o steps-info.json
[2020-07-07T16:48:07.716Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18706/runs/5/tests/?status=FAILED -o tests-errors.json
[2020-07-07T16:48:08.267Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18706/runs/5/log/ -o pipeline-log.txt

Settings struct {
Index struct {
Hidden string `json:"hidden"`
} `json:"index"`
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not Hidden bool, or Index IndexSettings json:"index"?

Copy link
Contributor Author

@ycombinator ycombinator Jul 7, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It had to do with the type of index.hidden in the Elasticsearch API's JSON response being a string, e.g. "true", instead of being a JSON boolean, e.g. true. Let me make a commit showing what I tried before this, implementing json.Unmarshaler, then we can decide which implementation we like better.

Copy link
Contributor Author

@ycombinator ycombinator Jul 7, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@urso I've made an alternate implementation (the one I tried originally) in 87b73dd. Let me know if you prefer that or the one right before that. Thanks!

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Uff, I see. If we would use something like boolstr more often we might consider to provide a package/library with decoding primitives. But here as a standalone single use case within the package I think I prefer to decode into string first. A hybrid solution could be Index struct { Hidden boolstr json:"index" }, but keep IndexSetting{ Hidden bool }. Then you might be able to cast from Index into IndexSetting directly. Using boolstr we see decoding errors directly returned from the JSON decoder if something changes.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, let me try the hybrid solution next.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@urso Implemented the hybrid approach in eb621b5. Let me know what you think. Thanks!

@ycombinator ycombinator force-pushed the mb-es-xp-index-mark-hidden branch from c07c160 to 87b73dd Compare July 7, 2020 13:56
@ycombinator
Copy link
Contributor Author

CI failures are unrelated. See #19740.

@ycombinator ycombinator merged commit 9d3f09c into elastic:master Jul 8, 2020
@ycombinator ycombinator deleted the mb-es-xp-index-mark-hidden branch July 8, 2020 14:05
@ycombinator ycombinator added needs_backport PR is waiting to be backported to other branches. v7.9.0 v8.0.0 labels Jul 8, 2020
ycombinator added a commit that referenced this pull request Jul 9, 2020
* Report index hidden setting in index stats

* Adding CHANGELOG entry

* Use expand_wildcards in tests as well

* Using json.Unmarshaler instead

* Hybrid approach
melchiormoulin pushed a commit to melchiormoulin/beats that referenced this pull request Oct 14, 2020
* Report index hidden setting in index stats

* Adding CHANGELOG entry

* Use expand_wildcards in tests as well

* Using json.Unmarshaler instead

* Hybrid approach
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Feature:Stack Monitoring needs_backport PR is waiting to be backported to other branches. Team:Services (Deprecated) Label for the former Integrations-Services team v7.9.0 v8.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Hidden indices not exposed in monitoring index_stats documents
4 participants