Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Infinite Tracing payload compression and batching #1146

Merged

Conversation

toddwest
Copy link
Contributor

@toddwest toddwest commented Feb 2, 2023

Overview

This PR adds support for Infinite Tracing payloads to be both compressed and batched by default. The following changes are included as part of this work:

  • Implements support for compressing infinite tracing payloads
  • gzip compression is enabled by default for batching and non-batching configurations
  • Implements support for batching spans within infinite tracing payloads
  • Defaults for batching are:
    • Batching enabled by default
    • Up to 100 spans bundled per batch
    • Waiting (linger) for up to 5s to fill up the batch before sending

The span data is still streaming over HTTP/2 using gRPC but utilizing compression and batching will significantly reduce payload sizes and bytes over the wire, resulting in significantly lower network transfer costs for customers.

Testing

All new functionality has been covered with unit tests where possible.

Checks

  • Are your contributions backwards compatible with relevant frameworks and APIs?

    • Yes
  • Does your code contain any breaking changes?

    • No (although infinite tracing defaults are changing)
  • Does your code introduce any new dependencies?

    • Only a testImplementation dependency for parameterized tests

…ng. Payloads being sent to trace observers will be compressed with `gzip` compression by default and will attempt to send batches of up to 100 spans by default. This replaces the existing default of no compression and no batching. The data is still streaming over HTTP/2 using gRPC but utilizing compression and batching to significantly reduce payload sizes and bytes over the wire, resulting in lower network transfer costs.
@jasonjkeller
Copy link
Contributor

Whoever reviews this PR, make sure that it is in alignment with the agent spec: https://source.datanerd.us/agents/agent-specs/pull/597

@kford-newrelic kford-newrelic added the on-roadmap Issue has been added to our product roadmap and will be worked in the coming quarter label Feb 10, 2023
@kford-newrelic
Copy link
Contributor

@toddwest thanks for the PR! We've got a number of things in our ToDo list that we have to take care of first but we're going to try and get this in this Jan-Mar quarter!

@kford-newrelic kford-newrelic removed the on-roadmap Issue has been added to our product roadmap and will be worked in the coming quarter label Feb 14, 2023
@kford-newrelic
Copy link
Contributor

@toddwest we had to bump this out to the Apr-Jun quarter since we're over-capacity for the current quarter

@CLAassistant
Copy link

CLAassistant commented Feb 17, 2023

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@toddwest toddwest force-pushed the towest/infinite-tracing-compression-batching branch 2 times, most recently from 5ec75ff to 302e103 Compare February 17, 2023 20:56
…bility metrics to bring implementation in line with agent spec.
@toddwest
Copy link
Contributor Author

Thanks for the reminder @jasonjkeller. Went ahead and updated to bring the PR in line with the spec.

@kford-newrelic I'm happy to spend an hour walking through the PR with the team if that could help expedite the review process? I think a lot of customers would get some significant relief from these changes.

@kford-newrelic kford-newrelic added the on-roadmap Issue has been added to our product roadmap and will be worked in the coming quarter label Mar 8, 2023
@jtduffy
Copy link
Contributor

jtduffy commented May 15, 2023

  • All PR tests passed from the branch I created from the fork.
  • Tested infinite tracing successfully

@jtduffy jtduffy merged commit 4f0ed92 into newrelic:main May 15, 2023
@toddwest toddwest deleted the towest/infinite-tracing-compression-batching branch May 16, 2023 22:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
on-roadmap Issue has been added to our product roadmap and will be worked in the coming quarter
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

6 participants