Test / Fix distributed trace across multiple AWS lambdas

[kford-newrelic]

Description

As a reference, the Python Agent team has a write up on what they needed to do to address this problem within the Python agent.

This topic came up as part of customer escalation, who was having some challenges with New Relic when using AWS API Gateway configured in HTTP mode which executes a lambda. For this customer, there are actually a chain of API Gateway (HTTP) → Lambda → API Gateway (HTTP) → Lambda calls. When manually testing this use case to understand the behavior within New Relic, it was found that each lambda had their own trace_id’s even though the expected HTTP headers were being forwarded to the second lambda.

Acceptance Criteria

If both entities have the same trace_id and the service map for the distributed trace shows both lambdas, you are done and no changes are needed.

If both entities have different trace_id’s and the service map for the distributed trace only shows the lambda that you are viewing from, then there is an issue with propagating the trace_id between the lambdas that needs to be debugged and fixed.

Design Consideration/Limitations

N/A

Dependencies

In order to run the test, it does take a bit of setup within AWS.

• Set up the “backend” API Gateway and Lambda

  • Create a lambda in Ruby that will return the contents of the incoming event as JSON as part of the response (backend lambda)
  • Create an API Gateway in HTTP mode (not REST) that will forward the request to the lambda you just created
  • Test for basic functionality
  • Update the lambda to use the appropriate New Relic lambda layer
  • Test and ensure the lambda entity is defined in New Relic as well as the distributed trace being available

• Set up the “frontend” API Gateway and Lambda

  • Create a lambda in Ruby that will make an HTTP request to the “backend” API Gateway URL. The response should include the event for the “frontend” lambda as well as the results reported by the “backend” call
  • Create an API Gateway in HTTP mode (not REST) that will forward the request to the lambda you just created
  • Test for basic functionality
  • Update the lambda to use the appropriate New Relic lambda layer
  • Test and ensure the lambda entity is defined in New Relic as well as the distributed trace being available

With the above defined and working, run curl commands to generate several test distributed traces. For the same external request, are both entities using the same trace_id or are they generating different trace_ids?

Additional context

See NR-265378 for Python example code and output

[workato-integration]

https://new-relic.atlassian.net/browse/NR-274393

[fallwith]

Addressed with #2811