Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid duplicate names in dynamic_templates #18849

Merged
merged 3 commits into from
Jun 2, 2020

Conversation

simitt
Copy link
Contributor

@simitt simitt commented May 29, 2020

What does this PR do?

When multiple dynamic templates are detected for the same field, add the
matching type to the template name for uniqueness.

Why is it important?

As described in #17203 (comment) the APM Server depends on allowing multiple dynamic templates for the same field, with different matching types. Therefore this PR takes care of adding some unique part to the name rather than failing with an error.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
    - [ ] I have made corresponding changes to the documentation
    - [ ] I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

This can be tested with APM Server as it creates multiple dynamic templates for the labels field.

  • Check out this branch and APM Server master and redirect to use this branch instead of the remote beats version. Can be achieved by running go mod edit -replace github.com/elastic/beats/v7=<path-to-your-local-beats-repo>.
  • start ES instance; compile and run APM Server make update apm-server && ./apm-server -e -E setup.template.overwrite=true
  • fetch the apm-server template apm-8.0.0 and search for labels_string, labels_boolean and labels_*.

Related issues

Screenshots

Screenshot 2020-05-29 at 15 54 04

When multiple dynamic templates are detected for the same field, add the
matching type to the template name for uniqueness.

closes elastic#17203
@simitt simitt added libbeat Team:Integrations Label for the Integrations team v7.9.0 labels May 29, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations (Team:Integrations)

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels May 29, 2020
@elasticmachine
Copy link
Collaborator

elasticmachine commented May 29, 2020

💔 Tests Failed

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #18849 updated]

  • Start Time: 2020-06-02T07:51:21.189+0000

  • Duration: 71 min 40 sec

Test stats 🧪

Test Results
Failed 6
Passed 9262
Skipped 1571
Total 10839

Test errors

Expand to view the tests failures

  • Name: Build and Test / Metricbeat x-pack / Metricbeat x-pack / TestFetch – channels

    • Age: 1
    • Duration: 10.11
    • Error Details: Failed
  • Name: Build and Test / Metricbeat x-pack / Metricbeat x-pack / TestFetch – stats

    • Age: 1
    • Duration: 12.54
    • Error Details: Failed
  • Name: Build and Test / Metricbeat x-pack / Metricbeat x-pack / TestFetch – subscriptions

    • Age: 1
    • Duration: 32.1
    • Error Details: Failed
  • Name: Build and Test / Metricbeat x-pack / Metricbeat x-pack / TestData – channels

    • Age: 4
    • Duration: 59.6
    • Error Details: Failed
  • Name: Build and Test / Metricbeat x-pack / Metricbeat x-pack / TestData – stats

    • Age: 4
    • Duration: 24.23
    • Error Details: Failed
  • Name: Build and Test / Metricbeat x-pack / Metricbeat x-pack / TestData – subscriptions

    • Age: 4
    • Duration: 48.56
    • Error Details: Failed

Steps errors

Expand to view the steps failures

  • Name: Mage build test
    • Description: mage build test

    • Duration: 30 min 52 sec

    • Start Time: 2020-06-02T08:16:55.943+0000

    • log

Log output

Expand to view the last 100 lines of log output

[2020-06-02T09:01:16.228Z]  Experimental:      false
[2020-06-02T09:01:16.228Z] Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
[2020-06-02T09:01:16.228Z] It requires Docker daemon to be installed and running
[2020-06-02T09:01:32.157Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849/src/github.com/elastic/beats
[2020-06-02T09:01:32.470Z] + find . -type f -name TEST*.xml -path */build/* -delete
[2020-06-02T09:01:32.483Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849/src/github.com/elastic/beats/Lint
[2020-06-02T09:01:32.575Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849/src/github.com/elastic/beats/Elastic-Agent-Mac-OS-X
[2020-06-02T09:01:32.661Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849/src/github.com/elastic/beats/Winlogbeat-oss
[2020-06-02T09:01:32.754Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849/src/github.com/elastic/beats/Elastic-Agent-x-pack
[2020-06-02T09:01:32.827Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849/src/github.com/elastic/beats/Auditbeat-crosscompile
[2020-06-02T09:01:32.917Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849/src/github.com/elastic/beats/Auditbeat-oss-Mac-OS-X
[2020-06-02T09:01:32.995Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849/src/github.com/elastic/beats/Dockerlogbeat
[2020-06-02T09:01:33.093Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849/src/github.com/elastic/beats/Filebeat-x-pack-Mac-OS-X
[2020-06-02T09:01:33.183Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849/src/github.com/elastic/beats/Journalbeat-oss
[2020-06-02T09:01:33.258Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849/src/github.com/elastic/beats/Generators-Metricbeat-Linux
[2020-06-02T09:01:33.335Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849/src/github.com/elastic/beats/Filebeat-Mac-OS-X
[2020-06-02T09:01:33.418Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849/src/github.com/elastic/beats/Metricbeat-Mac-OS-X
[2020-06-02T09:01:33.530Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849/src/github.com/elastic/beats/Functionbeat-x-pack
[2020-06-02T09:01:33.641Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849/src/github.com/elastic/beats/Auditbeat-x-pack-Mac-OS-X
[2020-06-02T09:01:33.722Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849/src/github.com/elastic/beats/Elastic-Agent-x-pack-Windows
[2020-06-02T09:01:33.795Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849/src/github.com/elastic/beats/Metricbeat-OSS-Unit-tests
[2020-06-02T09:01:33.867Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849/src/github.com/elastic/beats/Metricbeat-x-pack-Mac-OS-X
[2020-06-02T09:01:33.954Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849/src/github.com/elastic/beats/Metricbeat-crosscompile
[2020-06-02T09:01:34.048Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849/src/github.com/elastic/beats/Heartbeat-oss
[2020-06-02T09:01:34.129Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849/src/github.com/elastic/beats/Auditbeat-x-pack
[2020-06-02T09:01:34.209Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849/src/github.com/elastic/beats/Auditbeat-x-pack-Windows
[2020-06-02T09:01:34.297Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849/src/github.com/elastic/beats/Auditbeat-oss-Windows
[2020-06-02T09:01:34.393Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849/src/github.com/elastic/beats/Libbeat-x-pack
[2020-06-02T09:01:34.464Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849/src/github.com/elastic/beats/Winlogbeat-Windows-x-pack
[2020-06-02T09:01:34.561Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849/src/github.com/elastic/beats/Filebeat-Windows
[2020-06-02T09:01:34.649Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849/src/github.com/elastic/beats/Packetbeat-oss
[2020-06-02T09:01:34.736Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849/src/github.com/elastic/beats/Filebeat-x-pack-Windows
[2020-06-02T09:01:34.813Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849/src/github.com/elastic/beats/Auditbeat-oss-Linux
[2020-06-02T09:01:34.905Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849/src/github.com/elastic/beats/Functionbeat-Mac-OS-X-x-pack
[2020-06-02T09:01:34.987Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849/src/github.com/elastic/beats/Metricbeat-x-pack-Windows
[2020-06-02T09:01:35.060Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849/src/github.com/elastic/beats/Metricbeat-Windows
[2020-06-02T09:01:35.142Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849/src/github.com/elastic/beats/Winlogbeat-Windows
[2020-06-02T09:01:35.238Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849/src/github.com/elastic/beats/Filebeat-x-pack
[2020-06-02T09:01:35.321Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849/src/github.com/elastic/beats/Generators-Beat-Linux
[2020-06-02T09:01:35.467Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849/src/github.com/elastic/beats/Heartbeat-Mac-OS-X
[2020-06-02T09:01:35.559Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849/src/github.com/elastic/beats/Filebeat-oss
[2020-06-02T09:01:35.644Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849/src/github.com/elastic/beats/Functionbeat-Windows
[2020-06-02T09:01:35.720Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests
[2020-06-02T09:01:35.804Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849/src/github.com/elastic/beats/Metricbeat-x-pack
[2020-06-02T09:01:35.904Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849/src/github.com/elastic/beats/Metricbeat-Python-integration-tests
[2020-06-02T09:01:35.979Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849/src/github.com/elastic/beats/Libbeat-oss
[2020-06-02T09:01:36.055Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849/src/github.com/elastic/beats/Heartbeat-Windows
[2020-06-02T09:01:36.123Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849/src/github.com/elastic/beats/Generators-Metricbeat-Mac-OS-X
[2020-06-02T09:01:36.192Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849/src/github.com/elastic/beats/Libbeat-crosscompile
[2020-06-02T09:01:36.259Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849/src/github.com/elastic/beats/Libbeat-stress-tests
[2020-06-02T09:01:36.361Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849/src/github.com/elastic/beats/Generators-Beat-Mac-OS-X
[2020-06-02T09:01:36.734Z] + cat
[2020-06-02T09:01:36.734Z] + /usr/local/bin/runbld ./runbld-script
[2020-06-02T09:01:36.734Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-06-02T09:01:43.324Z] runbld>>> runbld started
[2020-06-02T09:01:43.324Z] runbld>>> 1.6.11/a66728ff8f4356963772e6e6d2069392fa06acbe
[2020-06-02T09:01:45.232Z] runbld>>> The following profiles matched the job 'Beats/beats-beats-mbp/PR-18849' in order of occurrence in the config (last value wins).
[2020-06-02T09:01:46.614Z] runbld>>> Debug logging enabled.
[2020-06-02T09:01:46.614Z] runbld>>> Storing result
[2020-06-02T09:01:46.614Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-06-02T09:01:46.614Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200602090146-1E44B5F3
[2020-06-02T09:01:46.614Z] runbld>>> Adding system facts.
[2020-06-02T09:01:47.556Z] runbld>>> Adding vcs info for the latest commit:  3a14b280fcfeff96ad9b4a9150ff3dc94b29ebd2
[2020-06-02T09:01:47.816Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-06-02T09:01:47.816Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-06-02T09:01:47.816Z] Processing JUnit reports with runbld...
[2020-06-02T09:01:47.816Z] + echo 'Processing JUnit reports with runbld...'
[2020-06-02T09:01:48.387Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-06-02T09:01:48.387Z] runbld>>> DURATION: 27ms
[2020-06-02T09:01:48.387Z] runbld>>> STDOUT: 40 bytes
[2020-06-02T09:01:48.387Z] runbld>>> STDERR: 49 bytes
[2020-06-02T09:01:48.387Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-06-02T09:01:48.387Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849/src/github.com/elastic/beats
[2020-06-02T09:01:49.768Z] runbld>>> Storing build metadata: 
[2020-06-02T09:01:49.768Z] runbld>>> Adding test report.
[2020-06-02T09:01:49.768Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849/src/github.com/elastic/beats
[2020-06-02T09:01:51.148Z] runbld>>> Found 114 test output files
[2020-06-02T09:01:51.148Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests/metricbeat/build/TEST-go-integration-graphite.xml
[2020-06-02T09:01:51.410Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests/metricbeat/build/TEST-go-integration-windows.xml
[2020-06-02T09:01:51.673Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-openmetrics.xml
[2020-06-02T09:01:51.673Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-cloudfoundry.xml
[2020-06-02T09:01:51.673Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-iis.xml
[2020-06-02T09:01:51.673Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-istio.xml
[2020-06-02T09:01:51.673Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-activemq.xml
[2020-06-02T09:01:51.673Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-tomcat.xml
[2020-06-02T09:01:53.585Z] runbld>>> Test output logs contained: Errors: 0 Failures: 6 Tests: 10689 Skipped: 1335
[2020-06-02T09:01:53.585Z] runbld>>> Storing result
[2020-06-02T09:01:53.585Z] runbld>>> FAILURES: 6
[2020-06-02T09:01:54.966Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-06-02T09:01:54.966Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200602090146-1E44B5F3
[2020-06-02T09:01:54.966Z] runbld>>> Email notification disabled by environment variable.
[2020-06-02T09:01:54.966Z] runbld>>> Slack notification disabled by environment variable.
[2020-06-02T09:02:00.438Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18849
[2020-06-02T09:02:00.536Z] [INFO] getVaultSecret: Getting secrets
[2020-06-02T09:02:00.582Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-06-02T09:02:01.260Z] + chmod 755 generate-build-data.sh
[2020-06-02T09:02:01.260Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18849/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18849/runs/3 FAILURE 4239808
[2020-06-02T09:02:01.260Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18849/runs/3/steps/?limit=10000 -o steps-info.json
[2020-06-02T09:02:03.122Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18849/runs/3/tests/?status=FAILED -o tests-errors.json
[2020-06-02T09:02:03.673Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18849/runs/3/log/ -o pipeline-log.txt

Copy link
Contributor

@exekias exekias left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change LGTM, thank you for submitting it! could you please add a changelog entry?

@simitt simitt requested a review from exekias June 2, 2020 09:10
@simitt
Copy link
Contributor Author

simitt commented Jun 2, 2020

test failures seem unrelated

@exekias exekias added the needs_backport PR is waiting to be backported to other branches. label Jun 2, 2020
@simitt simitt merged commit 4b9bc97 into elastic:master Jun 2, 2020
simitt added a commit to simitt/beats that referenced this pull request Jun 2, 2020
When multiple dynamic templates are detected for the same field, add the
matching type to the template name for uniqueness.

closes elastic#17203
simitt added a commit that referenced this pull request Jun 2, 2020
When multiple dynamic templates are detected for the same field, add the
matching type to the template name for uniqueness.

closes #17203
v1v added a commit to v1v/beats that referenced this pull request Jun 2, 2020
…-stage-level

* upstream/master:
  [CI] Fix permissions should not fail (elastic#18899)
  Revert "Allow the Docker image to be run with a random user id (elastic#12905)" (elastic#18872)
  Add new fields to HAProxy module of Metricbeat (elastic#18523)
  Avoid duplicate names in dynamic_templates (elastic#18849)
@simitt simitt deleted the dynamic-template-keys branch August 4, 2020 14:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
libbeat needs_backport PR is waiting to be backported to other branches. Team:Integrations Label for the Integrations team v7.9.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Check for duplicates in template generation Using same key in dynamic template leads to overwrites
4 participants