Introduce configuration for specifying service-specific source maps

[axw]

As part of the move to Fleet we will need to move away from fetching source maps directly from (and indexing source maps into) Elasticsearch, as the privileges APM Server is given does not cover this. Instead, source maps should be uploaded to Kibana, stored as a Fleet Artifact, with references to Fleet Artifact download URLs pushed down to APM Server via the server's policy.

In order to make the above possible, we will add configuration to APM Server to specify zero or more source maps, including:

• service name
• service version
• bundle filepath
• source map URL (typically a Fleet Server artifact download URL)

For example:

apm-server:
  rum:
    source_mapping:
      source_maps:
      - service.name: opbeans-rum
        service.version: 1.2.3
        bundle.filepath: /test/e2e/general-usecase/bundle.js.map
        sourcemap.url: http://somewhere.com/bundle.js.map

The server will fetch and cache these like today: fetch on first access, cache in memory with an expiration time. We might make improvements to caching later, but that can be done separately. If the list of source maps is dynamically updated, then the cache will be invalidated for (service name, service version, bundle filepath) tuples that are added, changed, or removed.

Configuring a list of source maps will disable fetching source maps from Elasticsearch. It will be an error to specify Elasticsearch configuration under apm-server.rum.source_mapping if the source_maps property is non-null.

Fleet Server requires auth for fetching artifacts. If a source map URL points to Fleet Server, then APM Server must pass its Fleet API access key in download requests.

[stuartnelson3]

Excepting cache management, is it sufficient to make a request to sourcemap.url with the auth header to get the sourcemap for a given key lookup? I'm not finding documentation, but the code seems to indicate that is enough:

https://github.com/elastic/fleet-server/blob/d72fcdd6d9c0707fef43918c2398b741cef77130/cmd/fleet/router.go#L72-L74
https://github.com/elastic/fleet-server/blob/d72fcdd6d9c0707fef43918c2398b741cef77130/cmd/fleet/handleArtifacts.go#L84

[axw]

@stuartnelson3 yes that should be sufficient - it should be enough to perform a GET request with the configured URL, with the API Key in the Authorization header.

[stuartnelson3]

If the list of source maps is dynamically updated, then the cache will be invalidated for (service name, service version, bundle filepath) tuples that are added, changed, or removed.

Looking at the way the apm-server is currently being reloaded when a config changed is pushed down, it's creating a new server on each reload. Unless I'm missing something, I think we would have to implement the reload differently in order to maintain still-valid sourcemaps in the cache

edit:

it appears go-cache has support for seeding the cache. if this feature is critical, we could do a best-effort load/unload of the cache to disk when the server is stopped.

https://pkg.go.dev/github.com/patrickmn/go-cache?utm_source=godoc#NewFrom

[stuartnelson3]

Is there coordination around applying sourcemaps? It looks like if multiple stacktraces come in at once that all perform a lookup on the same service name/version/bundlepath, there could be a storm of requests to the fleet-server (or, currently, elasticsearch). Is this a concern?

[axw]

Looking at the way the apm-server is currently being reloaded when a config changed is pushed down, it's creating a new server on each reload. Unless I'm missing something, I think we would have to implement the reload differently in order to maintain still-valid sourcemaps in the cache

Right, that's the current state of affair. A while back I opened #5039 to improve things.

it appears go-cache has support for seeding the cache. if this feature is critical, we could do a best-effort load/unload of the cache to disk when the server is stopped.

I think that would be sensible, but let's leave it to a followup.

[axw]

Is there coordination around applying sourcemaps? It looks like if multiple stacktraces come in at once that all perform a lookup on the same service name/version/bundlepath, there could be a storm of requests to the fleet-server (or, currently, elasticsearch). Is this a concern?

I don't recall what we have already, but I think it would be a good idea to to introduce to add some synchronisation: fetch it once, have others wait for the result.