[7.x] Add support for histograms to metrics intake (backport #5360) #5381

mergify · 2021-06-01T02:46:21Z

This is an automatic backport of pull request #5360 done by Mergify.
Cherry-pick of 0744428 has failed:

On branch mergify/bp/7.x/pr-5360
Your branch is up to date with 'origin/7.x'.

You are currently cherry-picking commit 07444287.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   beater/test_approved_es_documents/TestPublishIntegrationMetricsets.approved.json
	modified:   docs/spec/v2/metricset.json
	modified:   model/modeldecoder/generator/code.go
	modified:   model/modeldecoder/generator/jsonnumber.go
	modified:   model/modeldecoder/generator/jsonschema.go
	modified:   model/modeldecoder/generator/nstring.go
	modified:   model/modeldecoder/generator/slice.go
	modified:   model/modeldecoder/generator/validation.go
	modified:   model/modeldecoder/modeldecodertest/populator.go
	modified:   model/modeldecoder/rumv3/model_generated.go
	modified:   model/modeldecoder/v2/decoder.go
	modified:   model/modeldecoder/v2/metricset_test.go
	modified:   model/modeldecoder/v2/model.go
	modified:   model/modeldecoder/v2/model_generated.go
	modified:   processor/stream/test_approved_es_documents/testIntakeIntegrationMetricsets.approved.json
	modified:   processor/stream/test_approved_stream_result/testIntegrationResultMetricsets.approved.json
	modified:   systemtest/approvals/TestApprovedMetrics.approved.json
	modified:   systemtest/metrics_test.go
	modified:   systemtest/template_test.go
	modified:   testdata/intake-v2/metricsets.ndjson

Unmerged paths:
  (use "git add/rm <file>..." as appropriate to mark resolution)
	deleted by us:   changelogs/head.asciidoc

To fix up this pull request, you can check it out locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

Mergify commands and options

More conditions and actions can be found in the documentation.

You can also trigger Mergify actions by commenting on this pull request:

@Mergifyio refresh will re-evaluate the rules
@Mergifyio rebase will rebase this PR on its base branch
@Mergifyio update will merge the base branch into this PR
@Mergifyio backport <destination> will backport this PR on <destination> branch

Additionally, on Mergify dashboard you can:

look at your merge queues
generate the Mergify configuration with the config editor.

Finally, you can contact us on https://mergify.io/

* model/modeldecoder: add metric type and unit * systemtest: test histogram metrics * Update changelog * systemtest: fix min docs expectation in test (cherry picked from commit 0744428) # Conflicts: # changelogs/head.asciidoc

apmmachine · 2021-06-01T02:54:28Z

💔 Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #5381 updated
Start Time: 2021-06-01T02:51:18.541+0000
Duration: 43 min 4 sec
Commit: f626d42

Test stats 🧪

Test	Results
Failed	1
Passed	4705
Skipped	121
Total	4827

Trends 🧪

Test errors

Expand to view the tests failures

`Build and Test / System and Environment Tests / test_load_docs_with_template_and_add_metricset – tests.system.test_integration.Test`

Expand to view the error details

 beat.beat.TimeoutError: Timeout waiting for 'metric documents to reach 4' to be true. Waited 10 seconds.

Expand to view the stacktrace

 self = <test_integration.Test testMethod=test_load_docs_with_template_and_add_metricset>

    def test_load_docs_with_template_and_add_metricset(self):
>       self.load_docs_with_template(self.get_metricset_payload_path(), self.intake_url, 'metric', 4)

tests/system/test_integration.py:82: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/system/apmserver.py:288: in load_docs_with_template
    self.wait_for_events(endpoint, expected_events_count, index=query_index, max_timeout=max_timeout)
tests/system/apmserver.py:308: in wait_for_events
    name="{} documents to reach {}".format(processor_name, expected_count),
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cond = <function ElasticTest.wait_for_events.<locals>.get_docs at 0x7fdf22d37378>
max_timeout = 10, poll_interval = 0.25, name = 'metric documents to reach 4'

    def wait_until(cond, max_timeout=10, poll_interval=0.25, name="cond"):
        """
        Like beat.beat.wait_until but catches exceptions
        In a ElasticTest `cond` will usually be a query, and we need to keep retrying
         eg. on 503 response codes
        """
        assert callable(cond), "First argument of wait_until must be a function"
    
        start = datetime.now()
        while datetime.now()-start < timedelta(seconds=max_timeout):
            try:
                result = cond()
                if result:
                    return result
            except AttributeError as ex:
                raise ex
            except Exception as ex:
                pass
            time.sleep(poll_interval)
        raise TimeoutError("Timeout waiting for '{}' to be true. ".format(name) +
>                          "Waited {} seconds.".format(max_timeout))
E       beat.beat.TimeoutError: Timeout waiting for 'metric documents to reach 4' to be true. Waited 10 seconds.

tests/system/helper.py:40: TimeoutError

Steps errors

Expand to view the steps failures

`Windows build`

Took 2 min 16 sec . View more details on here

`Run Linux tests`

Took 18 min 1 sec . View more details on here
Description: ./.ci/scripts/linux-test.sh

`Test Sync`

Took 4 min 36 sec . View more details on here
Description: ./.ci/scripts/sync.sh

Log output

Expand to view the last 100 lines of log output

[2021-06-01T03:21:17.301Z] 5.31s call     tests/system/test_requests.py::ClientSideTest::test_ok
[2021-06-01T03:21:17.301Z] 5.30s call     tests/system/test_requests.py::CorsTest::test_ok
[2021-06-01T03:21:17.301Z] 5.12s call     tests/system/test_apikey_cmd.py::APIKeyCommandTest::test_info_by_name
[2021-06-01T03:21:17.301Z] 5.09s call     tests/system/test_auth.py::TestAPIKeyWithESConfig::test_backend_intake
[2021-06-01T03:21:17.301Z] 5.01s call     tests/system/test_integration_sourcemap.py::SourcemappingIntegrationTest::test_fetch_latest_of_multiple_sourcemaps
[2021-06-01T03:21:17.301Z] 5.00s call     tests/system/test_integration_acm.py::AgentConfigurationIntegrationTest::test_rum_disabled
[2021-06-01T03:21:17.301Z] 5.00s call     tests/system/test_jaeger.py::TestAuthTag::test_jaeger_authorized
[2021-06-01T03:21:17.301Z] 4.75s call     tests/system/test_integration.py::EnrichEventIntegrationTest::test_enrich_backend_event
[2021-06-01T03:21:17.301Z] 4.73s call     tests/system/test_integration_acm.py::AgentConfigurationIntegrationTest::test_config_requests
[2021-06-01T03:21:17.301Z] 4.69s call     tests/system/test_jaeger.py::Test::test_jaeger_auth_tag_removed
[2021-06-01T03:21:17.301Z] =========================== short test summary info ============================
[2021-06-01T03:21:17.301Z] FAILED tests/system/test_integration.py::Test::test_load_docs_with_template_and_add_metricset
[2021-06-01T03:21:17.301Z] ============= 1 failed, 133 passed, 1 warning in 380.46s (0:06:20) =============
[2021-06-01T03:21:17.561Z] make: *** [Makefile:91: system-tests] Error 1
[2021-06-01T03:21:18.502Z] Makefile:96: recipe for target 'docker-system-tests' failed
[2021-06-01T03:21:18.502Z] make: *** [docker-system-tests] Error 2
[2021-06-01T03:21:18.502Z] + cleanup
[2021-06-01T03:21:18.502Z] + rm -rf /tmp/tmp.V6CJuJ0t5r
[2021-06-01T03:21:18.502Z] + .ci/scripts/docker-get-logs.sh
[2021-06-01T03:21:20.667Z] Post stage
[2021-06-01T03:21:20.685Z] Running in /var/lib/jenkins/workspace/pm-server_apm-server-mbp_PR-5381/src/github.com/elastic/apm-server/build
[2021-06-01T03:21:20.715Z] Archiving artifacts
[2021-06-01T03:21:21.049Z] Recording test results
[2021-06-01T03:21:21.647Z] [Checks API] No suitable checks publisher found.
[2021-06-01T03:21:22.004Z] + tar --version
[2021-06-01T03:21:22.328Z] + tar --exclude=system-tests-linux-files.tgz -czf system-tests-linux-files.tgz system-tests
[2021-06-01T03:21:22.611Z] Archiving artifacts
[2021-06-01T03:21:23.038Z] Failed in branch System and Environment Tests
[2021-06-01T03:21:50.336Z] Running python tests
[2021-06-01T03:21:53.031Z] 2021/06/01 03:21:52 exec: go list -m
[2021-06-01T03:21:53.679Z] >> python test: Unit Testing
[2021-06-01T03:21:56.967Z] ============================= test session starts =============================
[2021-06-01T03:21:56.967Z] platform win32 -- Python 3.8.6, pytest-6.0.1, py-1.9.0, pluggy-0.13.1
[2021-06-01T03:21:56.967Z] rootdir: C:\Users\jenkins\workspace\pm-server_apm-server-mbp_PR-5381\src\github.com\elastic\apm-server
[2021-06-01T03:21:56.967Z] plugins: rerunfailures-9.0, timeout-1.3.4
[2021-06-01T03:21:56.967Z] timeout: 90.0s
[2021-06-01T03:21:56.967Z] timeout method: thread
[2021-06-01T03:21:56.967Z] timeout func_only: False
[2021-06-01T03:21:56.967Z] collected 134 items
[2021-06-01T03:21:56.967Z] 
[2021-06-01T03:21:56.967Z] tests\system\test_apikey_cmd.py sssss                                    [  3%]
[2021-06-01T03:22:33.126Z] tests\system\test_auth.py .sss                                           [  6%]
[2021-06-01T03:22:33.126Z] tests\system\test_export.py sss                                          [  8%]
[2021-06-01T03:22:33.126Z] tests\system\test_integration.py sssssssssssssssssssssss                 [ 26%]
[2021-06-01T03:22:33.126Z] tests\system\test_integration_acm.py sssssssss                           [ 32%]
[2021-06-01T03:22:33.126Z] tests\system\test_integration_sourcemap.py ssssssssssssss                [ 43%]
[2021-06-01T03:22:33.126Z] tests\system\test_jaeger.py sssss                                        [ 47%]
[2021-06-01T03:22:33.126Z] tests\system\test_pipelines.py sssssss                                   [ 52%]
[2021-06-01T03:23:54.622Z] tests\system\test_requests.py ...........................                [ 72%]
[2021-06-01T03:23:54.623Z] tests\system\test_setup_index_management.py sssssssssssssss              [ 83%]
[2021-06-01T03:23:54.623Z] tests\system\test_tls.py ssssssssssssssssssssss                          [100%]
[2021-06-01T03:23:54.623Z] 
[2021-06-01T03:23:54.623Z] ============================== warnings summary ===============================
[2021-06-01T03:23:54.623Z] c:\users\jenkins\appdata\local\temp\python-env\build\ve\windows\lib\site-packages\_pytest\junitxml.py:446
[2021-06-01T03:23:54.623Z]   c:\users\jenkins\appdata\local\temp\python-env\build\ve\windows\lib\site-packages\_pytest\junitxml.py:446: PytestDeprecationWarning: The 'junit_family' default value will change to 'xunit2' in pytest 6.0. See:
[2021-06-01T03:23:54.623Z]     https://docs.pytest.org/en/stable/deprecations.html#junit-family-default-value-change-to-xunit2
[2021-06-01T03:23:54.623Z]   for more information.
[2021-06-01T03:23:54.623Z]     _issue_warning_captured(deprecated.JUNIT_XML_DEFAULT_FAMILY, config.hook, 2)
[2021-06-01T03:23:54.623Z] 
[2021-06-01T03:23:54.623Z] -- Docs: https://docs.pytest.org/en/stable/warnings.html
[2021-06-01T03:23:54.623Z] - generated xml file: C:\Users\jenkins\workspace\pm-server_apm-server-mbp_PR-5381\src\github.com\elastic\apm-server\build\TEST-python-unit.xml -
[2021-06-01T03:23:54.623Z] ============================ slowest 20 durations =============================
[2021-06-01T03:23:54.623Z] 8.09s call     tests/system/test_requests.py::RateLimitTest::test_rate_limit_small_hit
[2021-06-01T03:23:54.623Z] 7.68s call     tests/system/test_requests.py::CorsTest::test_ok
[2021-06-01T03:23:54.623Z] 7.67s call     tests/system/test_requests.py::ClientSideTest::test_ok
[2021-06-01T03:23:54.623Z] 5.40s call     tests/system/test_auth.py::TestAccessDefault::test_full_access
[2021-06-01T03:23:54.623Z] 4.15s call     tests/system/test_requests.py::CorsTest::test_preflight_bad_headers
[2021-06-01T03:23:54.623Z] 4.12s call     tests/system/test_requests.py::RateLimitTest::test_multiple_ips_rate_limit_hit
[2021-06-01T03:23:54.623Z] 4.11s call     tests/system/test_requests.py::RateLimitTest::test_multiple_ips_rate_limit
[2021-06-01T03:23:54.623Z] 4.10s call     tests/system/test_requests.py::RateLimitTest::test_rate_limit_hit
[2021-06-01T03:23:54.623Z] 4.10s call     tests/system/test_requests.py::RateLimitTest::test_rate_limit
[2021-06-01T03:23:54.623Z] 3.60s call     tests/system/test_requests.py::RateLimitTest::test_rate_limit_only_metadata
[2021-06-01T03:23:54.623Z] 3.19s call     tests/system/test_requests.py::Test::test_gzip
[2021-06-01T03:23:54.623Z] 3.19s call     tests/system/test_requests.py::Test::test_bad_json
[2021-06-01T03:23:54.623Z] 3.18s call     tests/system/test_requests.py::Test::test_deflate
[2021-06-01T03:23:54.623Z] 3.17s call     tests/system/test_requests.py::Test::test_ok_verbose
[2021-06-01T03:23:54.623Z] 3.16s call     tests/system/test_requests.py::Test::test_ok
[2021-06-01T03:23:54.623Z] 3.16s call     tests/system/test_requests.py::Test::test_validation_fail
[2021-06-01T03:23:54.623Z] 3.16s call     tests/system/test_requests.py::Test::test_empty
[2021-06-01T03:23:54.623Z] 3.15s call     tests/system/test_requests.py::Test::test_healthcheck
[2021-06-01T03:23:54.623Z] 3.15s call     tests/system/test_requests.py::Test::test_not_existent
[2021-06-01T03:23:54.623Z] 3.15s call     tests/system/test_requests.py::CorsTest::test_no_origin
[2021-06-01T03:23:54.623Z] =========== 28 passed, 106 skipped, 1 warning in 111.74s (0:01:51) ============
[2021-06-01T03:23:54.623Z] >> python test: Unit Testing Complete
[2021-06-01T03:23:55.831Z] Post stage
[2021-06-01T03:23:55.857Z] Recording test results
[2021-06-01T03:23:56.653Z] [Checks API] No suitable checks publisher found.
[2021-06-01T03:34:17.867Z] [INFO] For detailed information see: https://apm-ci.elastic.co/job/apm-integration-tests-selector-mbp/job/7.x/848/display/redirect
[2021-06-01T03:34:18.604Z] Copied 18 artifacts from "APM Integration Test MBP Selector » 7.x" build number 848
[2021-06-01T03:34:19.854Z] Post stage
[2021-06-01T03:34:19.869Z] Recording test results
[2021-06-01T03:34:21.092Z] [Checks API] No suitable checks publisher found.
[2021-06-01T03:34:22.055Z] Running on Jenkins in /var/lib/jenkins/workspace/pm-server_apm-server-mbp_PR-5381
[2021-06-01T03:34:22.154Z] [INFO] getVaultSecret: Getting secrets
[2021-06-01T03:34:22.221Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-06-01T03:34:23.303Z] + chmod 755 generate-build-data.sh
[2021-06-01T03:34:23.303Z] + ./generate-build-data.sh https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-server/apm-server-mbp/PR-5381/ https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-server/apm-server-mbp/PR-5381/runs/2 FAILURE 2584489
[2021-06-01T03:34:23.303Z] INFO: curl https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-server/apm-server-mbp/PR-5381/runs/2/steps/?limit=10000 -o steps-info.json
[2021-06-01T03:34:23.854Z] INFO: curl https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-server/apm-server-mbp/PR-5381/runs/2/tests/?status=FAILED -o tests-errors.json
[2021-06-01T03:34:24.488Z] INFO: curl https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-server/apm-server-mbp/PR-5381/runs/2/log/ -o pipeline-log.txt

Add support for histograms to metrics intake (#5360)

e9ad19e

* model/modeldecoder: add metric type and unit * systemtest: test histogram metrics * Update changelog * systemtest: fix min docs expectation in test (cherry picked from commit 0744428) # Conflicts: # changelogs/head.asciidoc

mergify bot added the conflicts There is a conflict in the backported pull request label Jun 1, 2021

mergify bot assigned axw Jun 1, 2021

Delete head.asciidoc

f626d42

axw removed the conflicts There is a conflict in the backported pull request label Jun 1, 2021

axw mentioned this pull request Jun 1, 2021

tests/system: remove redundant metricset test #5383

Closed

axw merged commit c40a2ad into 7.x Jun 2, 2021

axw deleted the mergify/bp/7.x/pr-5360 branch June 2, 2021 00:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[7.x] Add support for histograms to metrics intake (backport #5360) #5381

[7.x] Add support for histograms to metrics intake (backport #5360) #5381

mergify bot commented Jun 1, 2021

apmmachine commented Jun 1, 2021 •

edited

Loading

Build stats

Test stats 🧪

Trends 🧪

`Build and Test / System and Environment Tests / test_load_docs_with_template_and_add_metricset – tests.system.test_integration.Test`

`Windows build`

`Run Linux tests`

`Test Sync`

[7.x] Add support for histograms to metrics intake (backport #5360) #5381

[7.x] Add support for histograms to metrics intake (backport #5360) #5381

Conversation

mergify bot commented Jun 1, 2021

apmmachine commented Jun 1, 2021 • edited Loading

💔 Tests Failed

Build stats

Test stats 🧪

Trends 🧪

Test errors

Build and Test / System and Environment Tests / test_load_docs_with_template_and_add_metricset – tests.system.test_integration.Test

Steps errors

Windows build

Run Linux tests

Test Sync

Log output

apmmachine commented Jun 1, 2021 •

edited

Loading

`Build and Test / System and Environment Tests / test_load_docs_with_template_and_add_metricset – tests.system.test_integration.Test`

`Windows build`

`Run Linux tests`

`Test Sync`