Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

processor/otel: record array resource attributes #5704

Merged
merged 3 commits into from
Jul 15, 2021

Conversation

axw
Copy link
Member

@axw axw commented Jul 14, 2021

Motivation/summary

If an unhandled array attribute is found in resource attributes, store that as a label like we do with span attributes.

Also, handle array attributes when translating to Elastic spans and not just transactions.

Checklist

- [ ] Update CHANGELOG.asciidoc
- [ ] Documentation has been updated

How to test these changes

Send a trace with OTLP with array attributes at both the resource and span level, and ensure they are recorded as labels.

Related issues

Fixes #5663

If an unhandled array attribute is found in resource
attributes, store that as a label like we do with span
attributes.
@axw axw added backport-7.14 Backport PR to the 7.14 branch backport-7.x labels Jul 14, 2021
@axw axw marked this pull request as draft July 14, 2021 09:29
@axw axw marked this pull request as ready for review July 14, 2021 09:35
@axw axw requested a review from a team July 14, 2021 09:35
@apmmachine
Copy link
Contributor

apmmachine commented Jul 14, 2021

💔 Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2021-07-15T00:27:15.080+0000

  • Duration: 45 min 43 sec

  • Commit: 5985f4d

Test stats 🧪

Test Results
Failed 2
Passed 5963
Skipped 14
Total 5979

Trends 🧪

Image of Build Times

Image of Tests

Test errors 2

Expand to view the tests failures

Build and Test / APM Integration Tests / test_concurrent_req_dotnet – tests.agent.test_dotnet
    Expand to view the error details

     AssertionError: queried for [('processor.event', 'transaction'), ('service.name', ['dotnetapp'])], expected 1000, got 999 
    

    Expand to view the stacktrace

     dotnet = <tests.fixtures.agents.Agent object at 0x7f03bd2d5b90>
    
        @pytest.mark.version
        @pytest.mark.dotnet
        def test_concurrent_req_dotnet(dotnet):
            foo = Concurrent.Endpoint(dotnet.foo.url,
                                      dotnet.app_name,
                                      ["foo"],
                                      "GET /foo")
    >       Concurrent(dotnet.apm_server.elasticsearch, [foo], iters=2).run()
    
    tests/agent/test_dotnet.py:28: 
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
    tests/agent/concurrent_requests.py:254: in run
        self.check_counts(it)
    tests/agent/concurrent_requests.py:138: in check_counts
        assert_count([("processor.event", "transaction"), ("service.name", service_names)], transactions_count)
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
    
    terms = [('processor.event', 'transaction'), ('service.name', ['dotnetapp'])]
    expected = 1000
    
        def assert_count(terms, expected):
            """wait a bit for doc count to reach expectation"""
            @timeout_decorator.timeout(max_wait)
            def check_count(mut_actual):
                while True:
                    rsp = self.es.count(index=self.index, body=self.elasticsearch.term_q(terms))
                    mut_actual[0] = rsp["count"]
                    if mut_actual[0] >= expected:
                        return
                    time.sleep(backoff)
        
            mut_actual = [-1]  # keep actual count in this mutable
            try:
                check_count(mut_actual)
            except timeout_decorator.TimeoutError:
                pass
            actual = mut_actual[0]
    >       assert actual == expected, err.format(terms, expected, actual)
    E       AssertionError: queried for [('processor.event', 'transaction'), ('service.name', ['dotnetapp'])], expected 1000, got 999
    
    tests/agent/concurrent_requests.py:132: AssertionError 
    

Build and Test / APM Integration Tests / test_conc_req_all_agents – tests.agent.test_multiple_agents
    Expand to view the error details

     AssertionError: queried for [('processor.event', 'transaction'), ('service.name', ['dotnetapp', 'dotnetapp', 'flaskapp', 'flaskapp', 'djangoapp', 'djangoapp', 'expressapp', 'expressapp', 'railsapp', 'railsapp', 'gonethttpapp', 'gonethttpapp', 'springapp', 'springapp'])], expected 7000, got 6999 
    

    Expand to view the stacktrace

     es = <tests.fixtures.es.es.<locals>.Elasticsearch object at 0x7f03bd2d5e50>
    apm_server = <tests.fixtures.apm_server.apm_server.<locals>.APMServer object at 0x7f03bd189ed0>
    flask = <tests.fixtures.agents.Agent object at 0x7f0394683310>
    django = <tests.fixtures.agents.Agent object at 0x7f0394683110>
    dotnet = <tests.fixtures.agents.Agent object at 0x7f03bd2d5b90>
    express = <tests.fixtures.agents.Agent object at 0x7f039463f190>
    rails = <tests.fixtures.agents.Agent object at 0x7f03b43767d0>
    go_nethttp = <tests.fixtures.agents.Agent object at 0x7f03b47e8390>
    java_spring = <tests.fixtures.agents.Agent object at 0x7f03b43a4890>
    
        def test_conc_req_all_agents(es, apm_server, flask, django, dotnet, express, rails, go_nethttp, java_spring):
            dotnet_f = Concurrent.Endpoint(dotnet.foo.url,
                                           dotnet.app_name,
                                           ["foo"],
                                           "GET /foo",
                                           events_no=500)
            dotnet_b = Concurrent.Endpoint(dotnet.bar.url,
                                           dotnet.app_name,
                                           ["bar", "extra"],
                                           "GET /bar",
                                           events_no=500)
            flask_f = Concurrent.Endpoint(flask.foo.url,
                                          flask.app_name,
                                          ["app.foo"],
                                          "GET /foo",
                                          events_no=500)
            flask_b = Concurrent.Endpoint(flask.bar.url,
                                          flask.app_name,
                                          ["app.bar", "app.extra"],
                                          "GET /bar",
                                          events_no=500)
            django_f = Concurrent.Endpoint(django.foo.url,
                                           django.app_name,
                                           ["foo.views.foo"],
                                           "GET foo.views.show",
                                           events_no=500)
            django_b = Concurrent.Endpoint(django.bar.url,
                                           django.app_name,
                                           ["bar.views.bar", "bar.views.extra"],
                                           "GET bar.views.show",
                                           events_no=500)
            express_f = Concurrent.Endpoint(express.foo.url,
                                            express.app_name,
                                            ["app.foo"],
                                            "GET /foo",
                                            events_no=500)
            express_b = Concurrent.Endpoint(express.bar.url,
                                            express.app_name,
                                            ["app.bar", "app.extra"],
                                            "GET /bar",
                                            events_no=500)
            rails_f = Concurrent.Endpoint(rails.foo.url,
                                          rails.app_name,
                                          ["ApplicationController#foo"],
                                          "ApplicationController#foo",
                                          events_no=500)
            rails_b = Concurrent.Endpoint(rails.bar.url,
                                          rails.app_name,
                                          ["ApplicationController#bar", "app.extra"],
                                          "ApplicationController#bar",
                                          events_no=500)
            go_nethttp_f = Concurrent.Endpoint(go_nethttp.foo.url,
                                               go_nethttp.app_name,
                                               ["foo"],
                                               "GET /foo",
                                               events_no=500)
            go_nethttp_b = Concurrent.Endpoint(go_nethttp.bar.url,
                                               go_nethttp.app_name,
                                               ["bar", "extra"],
                                               "GET /bar",
                                               events_no=500)
            java_spring_f = Concurrent.Endpoint(java_spring.foo.url,
                                                java_spring.app_name,
                                                ["foo"],
                                                "GreetingController#foo",
                                                events_no=500)
            java_spring_b = Concurrent.Endpoint(java_spring.bar.url,
                                                java_spring.app_name,
                                                ["bar", "extra"],
                                                "GreetingController#bar",
                                                events_no=500)
        
            Concurrent(es, [
                dotnet_f, dotnet_b,
                flask_f, flask_b,
                django_f, django_b,
                express_f, express_b,
                rails_b, rails_f,
                go_nethttp_f, go_nethttp_b,
                java_spring_f, java_spring_b,
    >       ], iters=1).run()
    
    tests/agent/test_multiple_agents.py:84: 
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
    tests/agent/concurrent_requests.py:254: in run
        self.check_counts(it)
    tests/agent/concurrent_requests.py:138: in check_counts
        assert_count([("processor.event", "transaction"), ("service.name", service_names)], transactions_count)
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
    
    terms = [('processor.event', 'transaction'), ('service.name', ['dotnetapp', 'dotnetapp', 'flaskapp', 'flaskapp', 'djangoapp', 'djangoapp', ...])]
    expected = 7000
    
        def assert_count(terms, expected):
            """wait a bit for doc count to reach expectation"""
            @timeout_decorator.timeout(max_wait)
            def check_count(mut_actual):
                while True:
                    rsp = self.es.count(index=self.index, body=self.elasticsearch.term_q(terms))
                    mut_actual[0] = rsp["count"]
                    if mut_actual[0] >= expected:
                        return
                    time.sleep(backoff)
        
            mut_actual = [-1]  # keep actual count in this mutable
            try:
                check_count(mut_actual)
            except timeout_decorator.TimeoutError:
                pass
            actual = mut_actual[0]
    >       assert actual == expected, err.format(terms, expected, actual)
    E       AssertionError: queried for [('processor.event', 'transaction'), ('service.name', ['dotnetapp', 'dotnetapp', 'flaskapp', 'flaskapp', 'djangoapp', 'djangoapp', 'expressapp', 'expressapp', 'railsapp', 'railsapp', 'gonethttpapp', 'gonethttpapp', 'springapp', 'springapp'])], expected 7000, got 6999
    
    tests/agent/concurrent_requests.py:132: AssertionError 
    

Steps errors 1

Expand to view the steps failures

Test Sync
  • Took 3 min 11 sec . View more details on here
  • Description: ./.ci/scripts/sync.sh

Log output

Expand to view the last 100 lines of log output

[2021-07-15T01:02:38.620Z] --- PASS: TestAPMServerMonitoring (1.21s)
[2021-07-15T01:02:38.620Z] === RUN   TestAPMServerMonitoringBuiltinUser
[2021-07-15T01:02:38.620Z] --- PASS: TestAPMServerMonitoringBuiltinUser (2.03s)
[2021-07-15T01:02:38.620Z] === RUN   TestAPMServerOnboarding
[2021-07-15T01:02:38.620Z] --- PASS: TestAPMServerOnboarding (4.55s)
[2021-07-15T01:02:38.620Z] === RUN   TestOTLPGRPCTraces
[2021-07-15T01:02:38.620Z] --- PASS: TestOTLPGRPCTraces (4.00s)
[2021-07-15T01:02:38.620Z] === RUN   TestOTLPGRPCMetrics
[2021-07-15T01:02:38.620Z] --- PASS: TestOTLPGRPCMetrics (5.51s)
[2021-07-15T01:02:38.620Z] === RUN   TestOTLPGRPCAuth
[2021-07-15T01:02:38.620Z] --- PASS: TestOTLPGRPCAuth (3.75s)
[2021-07-15T01:02:38.620Z] === RUN   TestOTLPClientIP
[2021-07-15T01:02:38.620Z] --- PASS: TestOTLPClientIP (5.28s)
[2021-07-15T01:02:38.620Z] === RUN   TestOTLPAnonymous
[2021-07-15T01:02:38.620Z] --- PASS: TestOTLPAnonymous (0.23s)
[2021-07-15T01:02:38.620Z] === RUN   TestOTLPRateLimit
[2021-07-15T01:02:38.620Z] --- PASS: TestOTLPRateLimit (0.28s)
[2021-07-15T01:02:38.620Z] === RUN   TestRUMXForwardedFor
[2021-07-15T01:02:38.620Z] --- PASS: TestRUMXForwardedFor (3.89s)
[2021-07-15T01:02:38.620Z] === RUN   TestRUMAllowServiceNames
[2021-07-15T01:02:38.620Z] --- PASS: TestRUMAllowServiceNames (0.16s)
[2021-07-15T01:02:38.620Z] === RUN   TestRUMRateLimit
[2021-07-15T01:02:38.620Z] --- PASS: TestRUMRateLimit (0.93s)
[2021-07-15T01:02:38.620Z] === RUN   TestRUMCORS
[2021-07-15T01:02:38.620Z] --- PASS: TestRUMCORS (0.20s)
[2021-07-15T01:02:38.620Z] === RUN   TestKeepUnsampled
[2021-07-15T01:02:38.620Z] === RUN   TestKeepUnsampled/false
[2021-07-15T01:02:38.620Z] === RUN   TestKeepUnsampled/true
[2021-07-15T01:02:38.620Z] --- PASS: TestKeepUnsampled (8.46s)
[2021-07-15T01:02:38.620Z]     --- PASS: TestKeepUnsampled/false (5.07s)
[2021-07-15T01:02:38.620Z]     --- PASS: TestKeepUnsampled/true (3.39s)
[2021-07-15T01:02:38.620Z] === RUN   TestKeepUnsampledWarning
[2021-07-15T01:02:38.620Z] --- PASS: TestKeepUnsampledWarning (4.61s)
[2021-07-15T01:02:38.620Z] === RUN   TestTailSampling
[2021-07-15T01:02:38.620Z]     sampling_test.go:135: waiting for 100 "parent" transactions
[2021-07-15T01:02:38.620Z]     sampling_test.go:135: waiting for 100 "child" transactions
[2021-07-15T01:02:38.620Z] --- PASS: TestTailSampling (5.51s)
[2021-07-15T01:02:38.620Z] === RUN   TestTailSamplingUnlicensed
[2021-07-15T01:02:38.620Z] 2021/07/15 01:01:36 Starting container id: cd8ecfe076c6 image: docker.elastic.co/elasticsearch/elasticsearch:8.0.0-2f008f4a-SNAPSHOT
[2021-07-15T01:02:38.620Z] 2021/07/15 01:01:37 Waiting for container id cd8ecfe076c6 image: docker.elastic.co/elasticsearch/elasticsearch:8.0.0-2f008f4a-SNAPSHOT
[2021-07-15T01:02:38.620Z] 2021/07/15 01:01:55 Container is ready id: cd8ecfe076c6 image: docker.elastic.co/elasticsearch/elasticsearch:8.0.0-2f008f4a-SNAPSHOT
[2021-07-15T01:02:38.620Z] --- PASS: TestTailSamplingUnlicensed (28.71s)
[2021-07-15T01:02:38.620Z] === RUN   TestRUMErrorSourcemapping
[2021-07-15T01:02:38.620Z] --- PASS: TestRUMErrorSourcemapping (4.09s)
[2021-07-15T01:02:38.620Z] === RUN   TestRUMSpanSourcemapping
[2021-07-15T01:02:38.620Z] --- PASS: TestRUMSpanSourcemapping (4.57s)
[2021-07-15T01:02:38.620Z] === RUN   TestDuplicateSourcemapWarning
[2021-07-15T01:02:38.620Z] --- PASS: TestDuplicateSourcemapWarning (4.16s)
[2021-07-15T01:02:38.620Z] === RUN   TestNoMatchingSourcemap
[2021-07-15T01:02:38.620Z] --- PASS: TestNoMatchingSourcemap (5.51s)
[2021-07-15T01:02:38.620Z] === RUN   TestFetchLatestSourcemap
[2021-07-15T01:02:38.620Z] --- PASS: TestFetchLatestSourcemap (4.61s)
[2021-07-15T01:02:38.620Z] === RUN   TestSourcemapCaching
[2021-07-15T01:02:38.620Z] --- PASS: TestSourcemapCaching (5.51s)
[2021-07-15T01:02:38.620Z] === RUN   TestIndexTemplateCoverage
[2021-07-15T01:02:38.620Z] --- PASS: TestIndexTemplateCoverage (4.90s)
[2021-07-15T01:02:38.620Z] PASS
[2021-07-15T01:02:38.620Z] ok  	github.com/elastic/apm-server/systemtest	400.936s
[2021-07-15T01:02:38.620Z] === RUN   TestAPMServer
[2021-07-15T01:02:38.620Z] 2021/07/15 00:55:55 Building apm-server...
[2021-07-15T01:02:38.620Z] 2021/07/15 00:55:57 Built /var/lib/jenkins/workspace/pm-server_apm-server-mbp_PR-5704/src/github.com/elastic/apm-server/apm-server
[2021-07-15T01:02:38.620Z] --- PASS: TestAPMServer (2.42s)
[2021-07-15T01:02:38.620Z] === RUN   TestUnstartedAPMServer
[2021-07-15T01:02:38.620Z] --- PASS: TestUnstartedAPMServer (0.00s)
[2021-07-15T01:02:38.620Z] === RUN   TestAPMServerStartTLS
[2021-07-15T01:02:38.620Z] --- PASS: TestAPMServerStartTLS (0.12s)
[2021-07-15T01:02:38.620Z] === RUN   TestExpvar
[2021-07-15T01:02:38.620Z] --- PASS: TestExpvar (0.10s)
[2021-07-15T01:02:38.620Z] PASS
[2021-07-15T01:02:38.620Z] ok  	github.com/elastic/apm-server/systemtest/apmservertest	2.653s
[2021-07-15T01:02:38.620Z] ?   	github.com/elastic/apm-server/systemtest/benchtest	[no test files]
[2021-07-15T01:02:38.620Z] ?   	github.com/elastic/apm-server/systemtest/cmd/apmbench	[no test files]
[2021-07-15T01:02:38.620Z] ?   	github.com/elastic/apm-server/systemtest/estest	[no test files]
[2021-07-15T01:02:38.620Z] ?   	github.com/elastic/apm-server/systemtest/fleettest	[no test files]
[2021-07-15T01:02:38.620Z] + cleanup
[2021-07-15T01:02:38.620Z] + rm -rf /tmp/tmp.Jkd9i7KDqA
[2021-07-15T01:02:38.880Z] + .ci/scripts/docker-get-logs.sh
[2021-07-15T01:02:40.810Z] It is not possible to grab the logs of ef4dd9013fa8
[2021-07-15T01:02:41.986Z] Post stage
[2021-07-15T01:02:42.001Z] Running in /var/lib/jenkins/workspace/pm-server_apm-server-mbp_PR-5704/src/github.com/elastic/apm-server/build
[2021-07-15T01:02:42.027Z] Archiving artifacts
[2021-07-15T01:02:42.375Z] Recording test results
[2021-07-15T01:02:43.158Z] [Checks API] No suitable checks publisher found.
[2021-07-15T01:02:43.520Z] + tar --version
[2021-07-15T01:02:43.882Z] + tar --exclude=system-tests-linux-files.tgz -czf system-tests-linux-files.tgz system-tests
[2021-07-15T01:02:43.913Z] Archiving artifacts
[2021-07-15T01:02:44.355Z] Terminated
[2021-07-15T01:02:44.339Z] Terminated
[2021-07-15T01:11:51.198Z] [INFO] For detailed information see: https://apm-ci.elastic.co/job/apm-integration-tests-selector-mbp/job/master/18759/display/redirect
[2021-07-15T01:11:54.652Z] Copied 28 artifacts from "APM Integration Test MBP Selector » master" build number 18759
[2021-07-15T01:11:55.910Z] Post stage
[2021-07-15T01:11:55.927Z] Recording test results
[2021-07-15T01:11:56.805Z] [Checks API] No suitable checks publisher found.
[2021-07-15T01:11:57.228Z] Running on worker-1225339 in /var/lib/jenkins/workspace/pm-server_apm-server-mbp_PR-5704
[2021-07-15T01:11:57.294Z] [INFO] getVaultSecret: Getting secrets
[2021-07-15T01:11:57.347Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-07-15T01:11:59.696Z] + chmod 755 generate-build-data.sh
[2021-07-15T01:11:59.697Z] + ./generate-build-data.sh https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-server/apm-server-mbp/PR-5704/ https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-server/apm-server-mbp/PR-5704/runs/7 UNSTABLE 2683228
[2021-07-15T01:11:59.697Z] INFO: curl https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-server/apm-server-mbp/PR-5704/runs/7/steps/?limit=10000 -o steps-info.json
[2021-07-15T01:12:00.395Z] INFO: curl https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-server/apm-server-mbp/PR-5704/runs/7/tests/?status=FAILED -o tests-errors.json

@axw axw added the test-plan label Jul 14, 2021
@axw
Copy link
Member Author

axw commented Jul 14, 2021

/test

1 similar comment
@axw
Copy link
Member Author

axw commented Jul 14, 2021

/test

@axw
Copy link
Member Author

axw commented Jul 15, 2021

/test

@axw axw enabled auto-merge (squash) July 15, 2021 00:26
@axw axw disabled auto-merge July 15, 2021 02:19
@axw axw merged commit 4264a2a into elastic:master Jul 15, 2021
@axw axw deleted the otel-array-resource-attributes branch July 15, 2021 02:19
mergify bot pushed a commit that referenced this pull request Jul 15, 2021
* processor/otel: record array resource attributes

If an unhandled array attribute is found in resource
attributes, store that as a label like we do with span
attributes.

* processor/otel: handle array attrs for spans

(cherry picked from commit 4264a2a)
mergify bot pushed a commit that referenced this pull request Jul 15, 2021
* processor/otel: record array resource attributes

If an unhandled array attribute is found in resource
attributes, store that as a label like we do with span
attributes.

* processor/otel: handle array attrs for spans

(cherry picked from commit 4264a2a)
axw added a commit that referenced this pull request Jul 15, 2021
* processor/otel: record array resource attributes

If an unhandled array attribute is found in resource
attributes, store that as a label like we do with span
attributes.

* processor/otel: handle array attrs for spans

(cherry picked from commit 4264a2a)

Co-authored-by: Andrew Wilkins <[email protected]>
axw added a commit that referenced this pull request Jul 15, 2021
* processor/otel: record array resource attributes

If an unhandled array attribute is found in resource
attributes, store that as a label like we do with span
attributes.

* processor/otel: handle array attrs for spans

(cherry picked from commit 4264a2a)

Co-authored-by: Andrew Wilkins <[email protected]>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
@stuartnelson3 stuartnelson3 self-assigned this Jul 16, 2021
@stuartnelson3
Copy link
Contributor

Confirmed against SNAPSHOT

@stuartnelson3 stuartnelson3 removed their assignment Jul 20, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

OTel array attributes not recorded as labels
3 participants