Service specific source maps #5410

stuartnelson3 · 2021-06-03T11:09:40Z

Motivation/summary

support configuring source maps from kibana and fetching them from fleet server.

Checklist

Update CHANGELOG.asciidoc
Documentation has been updated

How to test these changes

Run elasticsearch + and elastic-agent in fleet mode.

Start kibana with the code from elastic/kibana#95393 has been completed, and apm-server with the fleet-server sourcemap code.

Upload a sourcemap via kibana
Confirm an updated config with the new source_maps block has been pushed to apm-server
Confirm ingesting rum traces for the uploaded sourcemap get mapped appropriately

Related issues

depends on elastic/kibana#95393

apmmachine · 2021-06-03T13:48:26Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #5410 updated
Start Time: 2021-06-23T09:36:35.727+0000
Duration: 53 min 25 sec
Commit: d098115

Test stats 🧪

Test	Results
Failed	0
Passed	6161
Skipped	119
Total	6280

Trends 🧪

Re-use the RoundTripper configuration code for the es client to configure the fleet http client the same way. Internally, the es client is instrumented, so do that as well.

axw

Approach looks good. The biggest unknown for me is where we get TLS config from for communicating with fleet-server.

beater/config/rum.go

sourcemap/store.go

sourcemap/fleet_store.go

beater/config/sourcemapping.go

beater/beater.go

this was a temporary fix

if a series of requests come in at once for a key that isn't in the cache, coordinate a single request to the backend to retrieve it and have the others wait until the request is complete.

sourcemap/store.go

simitt

Left a first high-level review.

_meta/beat.yml

beater/beater.go

sourcemap/store.go

Limit the number of open requests that are waiting for a response from the sourcemap backend

we won't be re-using elasticsearch config for fleet-server configuration.

mergify · 2021-06-08T03:23:50Z

This pull request is now in conflicts. Could you fix it @stuartnelson3? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b service-specific-source-maps upstream/service-specific-source-maps
git merge upstream/master
git push upstream service-specific-source-maps

Remove the TestDataStreams tests, which test the server's internal/private/undocumneted configuration for enabling data streams. At least for now, this is only intended to be used with Fleet, so we should instead just rely on the Fleet system test. There was a bug in the Fleet test where apm-server would create "traces-apm-default" as a plain old index instead of a data stream in certain conditions: if the apm integration had previously been installed, then the traces-apm template would be deleted by CleanupElasticsearch. Because the package was already installed, creating the policy would not automatically reinstall it and add the missing template. We fix the Fleet issue by uninstalling the package during Fleet cleanup, after unenrolling agents.

stuartnelson3 · 2021-06-22T15:02:15Z

@axw let me know if you spot anything else when you get the chance

axw

just a few more minor comments, otherwise LGTM!

beater/beater_test.go

sourcemap/fleet_store.go

sourcemap/fleet_store_test.go

stuartnelson3 · 2021-06-23T09:33:49Z

jenkins run the tests

Serve service-specific sourcemaps from fleet-server (cherry picked from commit 9453fe2) # Conflicts: # apmpackage/apm/manifest.yml # changelogs/head.asciidoc

* Service specific source maps (#5410) Serve service-specific sourcemaps from fleet-server (cherry picked from commit 9453fe2) # Conflicts: # apmpackage/apm/manifest.yml # changelogs/head.asciidoc * Update manifest.yml * Delete head.asciidoc Co-authored-by: stuart nelson <[email protected]>

Serve service-specific sourcemaps from fleet-server (cherry picked from commit 9453fe2) # Conflicts: # apmpackage/apm/manifest.yml # beater/beater.go # beater/beater_test.go # changelogs/head.asciidoc # sourcemap/fleet_store.go # sourcemap/fleet_store_test.go # sourcemap/store_test.go # tests/system/test_integration_sourcemap.py

axw · 2021-07-16T03:33:18Z

I tested this with the current Kibana API which requires sending the source map as a string, rather than as a file.

It works, but there's a small issue. I essentially manually ran this system test, but using the Kibana API for uploading the source map:

apm-server/systemtest/sourcemap_test.go

Lines 38 to 60 in 29cfed3

    
           func TestRUMErrorSourcemapping(t *testing.T) { 
        
           	systemtest.CleanupElasticsearch(t) 
        
           	srv := apmservertest.NewUnstartedServer(t) 
        
           	srv.Config.RUM = &apmservertest.RUMConfig{Enabled: true} 
        
           	err := srv.Start() 
        
           	require.NoError(t, err) 
        
           	uploadSourcemap(t, srv, "../testdata/sourcemap/bundle.js.map", 
        
           		"http://localhost:8000/test/e2e/../e2e/general-usecase/bundle.js.map", 
        
           		"apm-agent-js", 
        
           		"1.0.1", 
        
           	) 
        
           	systemtest.Elasticsearch.ExpectDocs(t, "apm-*-sourcemap", nil) 
        
           	systemtest.SendRUMEventsPayload(t, srv, "../testdata/intake-v2/errors_rum.ndjson") 
        
           	result := systemtest.Elasticsearch.ExpectDocs(t, "apm-*-error", nil) 
        
           	systemtest.ApproveEvents( 
        
           		t, t.Name(), result.Hits.Hits, 
        
           		// RUM timestamps are set by the server based on the time the payload is received. 
        
           		"@timestamp", "timestamp.us", 
        
           	) 
        
           }

The issue is that the "bundle_filepath" is apparently not being cleaned. The path with "e2e/../e2e" in it does not match, but when I manually clean it (i.e. change that substring to "e2e"), then it does match.

The existing APM Server endpoint cleans the URL path before storing it in Elasticsearch:

apm-server/beater/api/asset/sourcemap/handler.go

Line 130 in 29cfed3

smap.BundleFilepath = utility.CleanUrlPath(req.FormValue("bundle_filepath"))

Kibana should probably do the same. Alternatively we could clean it after receiving it in the APM Server, but seems better to store it cleaned in the first place. I'll open a Kibana issue.

axw · 2021-07-19T01:20:33Z

Calling this one working as expected, there's elastic/kibana#105911 open to sort out the issue noted above.

stuartnelson3 added 5 commits June 1, 2021 16:26

wip

2d72bd9

fleet/es branching working

39344ed

add apikey to fleetStore

d2e4248

[sourcemap/store] comment about duplicated config

ae01674

implement fleet store+test

5b8d875

stuartnelson3 added the v7.14.0 label Jun 3, 2021

Merge branch 'master' into service-specific-source-maps

5a6a1ba

stuartnelson3 added 2 commits June 3, 2021 17:22

[store/fleet] use same Transport as es client

b612d3f

Re-use the RoundTripper configuration code for the es client to configure the fleet http client the same way. Internally, the es client is instrumented, so do that as well.

Merge branch 'master' into service-specific-source-maps

c1a9d49

axw reviewed Jun 4, 2021

View reviewed changes

stuartnelson3 added 6 commits June 4, 2021 10:21

remove old sourcemap api key code

382e5dd

this was a temporary fix

use key struct for fleet config cache

d100bc1

flatten sourcemap config struct defn

3ccda4c

public elasticsearch/fleet constructor methods

679f2b5

synchronize cache fetch on miss

c3a8aa8

if a series of requests come in at once for a key that isn't in the cache, coordinate a single request to the backend to retrieve it and have the others wait until the request is complete.

write TODO about killing piling up requests

c082b70

stuartnelson3 commented Jun 4, 2021

View reviewed changes

sourcemap/store.go Show resolved Hide resolved

simitt reviewed Jun 7, 2021

View reviewed changes

_meta/beat.yml Outdated Show resolved Hide resolved

beater/beater.go Outdated Show resolved Hide resolved

sourcemap/store.go Outdated Show resolved Hide resolved

sourcemap/store.go Show resolved Hide resolved

stuartnelson3 added 6 commits June 7, 2021 11:15

Add waiters semaphore

c4d2b1f

Limit the number of open requests that are waiting for a response from the sourcemap backend

remove unnecessary else block

22dd903

add timeout for waiting

c42479e

update docs

9bd9bfe

update changelog

f112d11

stop configuring fleet-server http client

37e0d4a

we won't be re-using elasticsearch config for fleet-server configuration.

stuartnelson3 marked this pull request as ready for review June 7, 2021 15:37

stuartnelson3 added 2 commits June 8, 2021 09:23

/s/sourcemaps/metadata/

72a13fe

Merge branch 'master' into service-specific-source-maps

543f37b

stuartnelson3 marked this pull request as draft June 8, 2021 11:57

axw and others added 2 commits June 22, 2021 10:37

remove sourcemap api key from manifest

1cc5077

stuartnelson3 mentioned this pull request Jun 22, 2021

[sourcemaps] support multiple fleet addresses #5514

Closed

stuartnelson3 added 5 commits June 22, 2021 11:26

use single fleetcfg nil check

bb2c6c9

attach context to outgoing fleet request

65ad079

test sourcemap url path

7711ee4

Merge branch 'master' into service-specific-source-maps

3799cd5

remove duplicate rum enable field

051baa8

axw approved these changes Jun 23, 2021

View reviewed changes

beater/beater_test.go Show resolved Hide resolved

sourcemap/fleet_store.go Outdated Show resolved Hide resolved

sourcemap/fleet_store_test.go Outdated Show resolved Hide resolved

stuartnelson3 added 4 commits June 23, 2021 09:23

update fleet hosts TODO

85193f5

verify that fleet store correctly uses Protocol

77fc0a0

show fleet used if fleetcfg != nil

35468ac

Merge branch 'master' into service-specific-source-maps

78d2f1f

Merge branch 'master' into service-specific-source-maps

d098115

stuartnelson3 merged commit 9453fe2 into elastic:master Jun 23, 2021

stuartnelson3 deleted the service-specific-source-maps branch June 23, 2021 11:27

mergify bot pushed a commit that referenced this pull request Jun 23, 2021

Service specific source maps (#5410)

b976c32

Serve service-specific sourcemaps from fleet-server (cherry picked from commit 9453fe2) # Conflicts: # apmpackage/apm/manifest.yml # changelogs/head.asciidoc

mergify bot mentioned this pull request Jun 23, 2021

[7.x] Service specific source maps (backport #5410) #5524

Merged

stuartnelson3 added the test-plan label Jun 29, 2021

axw self-assigned this Jul 12, 2021

mergify bot mentioned this pull request Jul 12, 2021

[7.14] Service specific source maps (backport #5410) #5680

Closed

axw mentioned this pull request Jul 16, 2021

[APM] source map bundle_filepath is not cleaned elastic/kibana#105911

Closed

axw added the test-plan-ok label Jul 19, 2021

estolfo mentioned this pull request Jul 20, 2021

[sourcemaps] Support multiple fleet addresses when requesting a sourcemap #5770

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Service specific source maps #5410

Service specific source maps #5410

stuartnelson3 commented Jun 3, 2021 •

edited

Loading

apmmachine commented Jun 3, 2021 •

edited

Loading

Build stats

Test stats 🧪

Trends 🧪

axw left a comment

simitt left a comment

mergify bot commented Jun 8, 2021

stuartnelson3 commented Jun 22, 2021

axw left a comment

stuartnelson3 commented Jun 23, 2021

axw commented Jul 16, 2021

axw commented Jul 19, 2021

Service specific source maps #5410

Service specific source maps #5410

Conversation

stuartnelson3 commented Jun 3, 2021 • edited Loading

Motivation/summary

Checklist

How to test these changes

Related issues

apmmachine commented Jun 3, 2021 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

axw left a comment

Choose a reason for hiding this comment

simitt left a comment

Choose a reason for hiding this comment

mergify bot commented Jun 8, 2021

stuartnelson3 commented Jun 22, 2021

axw left a comment

Choose a reason for hiding this comment

stuartnelson3 commented Jun 23, 2021

axw commented Jul 16, 2021

axw commented Jul 19, 2021

stuartnelson3 commented Jun 3, 2021 •

edited

Loading

apmmachine commented Jun 3, 2021 •

edited

Loading