Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

the agent is sometimes setting the 'traceparent' header on requests to APM server #1995

Closed
trentm opened this issue Mar 11, 2021 · 0 comments
Assignees
Labels
agent-nodejs Make available for APM Agents project planning.

Comments

@trentm
Copy link
Member

trentm commented Mar 11, 2021

Describe the bug

Gil observed in the edge dev setup of opbeans that there were requests from the Node.js APM agent to apm-server's intake API that included the traceparent headers, for example:

screenshot of span details in Kibana showing traceparent

This makes the real transaction(s) with that trace id (in this case the opbeans-node requests) look like this apm-server intake API call is part of its distributed trace, which is bogus.

To Reproduce

Don't have a repro yet. For this to happen the agent would have had to have successfully created a span here: https://github.com/elastic/apm-agent-nodejs/blob/v3.12.1/lib/instrumentation/http-shared.js#L135
The theory is that the APM agent's request to apm-server happened in the context of a user transaction, and it piggybacked on that transaction, creating a span.

Ultimately, I believe, this code results in that span not being fully captured and sent. However the code adding the 'traceparent' header is above that.

Expected behavior

Requests to APM server should not get the 'traceparent' or 'tracestate' headers assigned.
Backing up a bit, an appropriate fix might be to see if we can ensure we don't get instrumentation interception of any of the APM agent's internal workings. This would avoid some log output confusion for users from this that appears for every request to APM server:

no active transaction found - cannot build new span
intercepted call to http.get { id: null }

Environment (please complete the following information)

  • OS: Linux (edge oblt dev setup)
  • Node.js and Agent version (from user-agent): elasticapm-node/3.10.0 elastic-apm-http-client/9.4.2 node/14.16.0
  • APM Server version: a recent 8.0.0
@trentm trentm changed the title the agent is sometimes set the 'traceparent' header on requests to APM server the agent is sometimes setting the 'traceparent' header on requests to APM server Mar 11, 2021
@github-actions github-actions bot added the agent-nodejs Make available for APM Agents project planning. label Mar 11, 2021
trentm added a commit to elastic/apm-nodejs-http-client that referenced this issue Aug 10, 2021
Use uninstrumented https.request() et al (grabbing references before the
agent does shimming) for requests to APM server. This avoids them
interfering in tracing (adding trace-context headers to APM server
requests, emitting debug logs about spans). This will fix
elastic/apm-agent-nodejs#1995
@trentm trentm self-assigned this Aug 10, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
agent-nodejs Make available for APM Agents project planning.
Projects
None yet
Development

No branches or pull requests

1 participant