feat(otlp-exporter): support `user-agent` headers in web exporters #3811

llc1123 · 2023-05-17T08:50:15Z

Which problem is this PR solving?

Support setting user-agent header in web exporters.

Fixes #3808

Update:

Will replace xhr with fetch in another PR.

Short description of the changes

Add user-agent header in web exporters in base packages.
Remove the use of sendBeacon which doesn't support custom headers.
Move the action of setting the user-agent header into base packages.
Warn the user if they try to config the user agent by config.
Correct the applying order of headers/metadata.
Normalize HTTP headers to lower case to prevent duplicate headers because they are case insensitive.
Fix a bug that env vars overwrite config from constructor parameters

Type of change

Please delete options that are not relevant.

New feature (non-breaking change which adds functionality)

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

Existing Tests

Checklist:

Followed the style guidelines of this project
Unit tests have been added

codecov · 2023-05-17T09:08:29Z

Codecov Report

Merging #3811 (bdab866) into main (2fb1b30) will decrease coverage by 0.04%.
The diff coverage is 93.33%.

❗ Current head bdab866 differs from pull request most recent head f3f0ffd. Consider uploading reports for the commit f3f0ffd to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3811      +/-   ##
==========================================
- Coverage   92.95%   92.92%   -0.04%     
==========================================
  Files         297      297              
  Lines        9060     9068       +8     
  Branches     1848     1852       +4     
==========================================
+ Hits         8422     8426       +4     
- Misses        638      642       +4

Impacted Files	Coverage Δ
...grpc-exporter-base/src/OTLPGRPCExporterNodeBase.ts	`72.91% <75.00%> (-0.90%)`	⬇️
...ter-base/src/platform/node/OTLPExporterNodeBase.ts	`51.61% <80.00%> (+1.61%)`	⬆️
...ges/exporter-logs-otlp-grpc/src/OTLPLogExporter.ts	`93.10% <100.00%> (+0.79%)`	⬆️
.../exporter-trace-otlp-grpc/src/OTLPTraceExporter.ts	`93.10% <100.00%> (+0.24%)`	⬆️
...e-otlp-http/src/platform/node/OTLPTraceExporter.ts	`100.00% <100.00%> (ø)`
...-otlp-proto/src/platform/node/OTLPTraceExporter.ts	`100.00% <100.00%> (ø)`
...porter-metrics-otlp-grpc/src/OTLPMetricExporter.ts	`93.75% <100.00%> (+0.20%)`	⬆️
...-otlp-http/src/platform/node/OTLPMetricExporter.ts	`100.00% <100.00%> (ø)`
...orter-metrics-otlp-proto/src/OTLPMetricExporter.ts	`100.00% <100.00%> (ø)`
...perimental/packages/otlp-exporter-base/src/util.ts	`93.33% <100.00%> (+0.22%)`	⬆️
... and 1 more

... and 1 file with indirect coverage changes

…tric exporters (open-telemetry#3806)" This reverts commit 6a876a0.

…ce exporters (open-telemetry#3790)" This reverts commit 758c7af.

…move `sendBeacon` which does not support custom headers

…nstructor parameter

experimental/packages/exporter-trace-otlp-grpc/test/OTLPTraceExporter.test.ts

JamieDanielson

I added my suggestion inline in case my previous comment wasn't clear. What do you think?

experimental/packages/exporter-trace-otlp-grpc/test/OTLPTraceExporter.test.ts

dyladan

I'd like to see some duplication reduction and the beacon sending should not be removed in this PR.

I also think this PR is quite large and each changelog entry should have its own PR so that the changes can be reviewed independently.

dyladan · 2023-05-31T14:54:48Z

experimental/CHANGELOG.md

 * feat(opencensus-shim): add OpenCensus trace shim [#3809](https://github.com/open-telemetry/opentelemetry-js/pull/3809) @aabmass

 ### :bug: (Bug Fix)

 * fix(sdk-node): use resource interface instead of concrete class [#3803](https://github.com/open-telemetry/opentelemetry-js/pull/3803) @blumamir
 * fix(sdk-logs): remove includeTraceContext configuration and use LogRecord context when available  [#3817](https://github.com/open-telemetry/opentelemetry-js/pull/3817) @hectorhdzg
+* fix(otlp-exporters): correct applying order of http headers/gRPC metadata [#3811](https://github.com/open-telemetry/opentelemetry-js/pull/3811) @llc1123


This changelog doesn't really describe what changed. Also, should this change be in the same PR or a separate one?

If the applying order of headers/metadata is not fixed, the implementation of user-agent header in base packages will not work. That's why #3790 #3806 chose to add the header in separate packages.

We had a discussion here.

experimental/packages/exporter-trace-otlp-grpc/src/OTLPTraceExporter.ts

dyladan · 2023-05-31T15:07:22Z

experimental/packages/otlp-exporter-base/src/platform/browser/OTLPExporterBrowserBase.ts

+      sendWithXhr(
+        body,
+        this.url,
+        this._headers,
+        this.timeoutMillis,
+        resolve,
+        reject
+      );


This is a breaking change. Beacon sending is an important feature that can't be removed without significant reasoning and discussion

The user-agent header is required, and sendBeacon() doesn't support HTTP headers at all. Also, the sendBeacon function is not used anywhere else. I think it should be safe removing it.

This is an environment specific requirement and there is precedent for breaking from spec for such reasons. Without beacon sending telemetry can be lost in many cases.

I do know the advantages of sendBeacon. But the user-agent header is required. The sendBeacon function will never be reached.

Using fetch with keepalive=true also allows the request to outlive the page. If we can replace xhr with fetch.

fetch with keep-alive is not supported in all environments, so we can't rely on simply using it instead.
While (most) current versions of environments that support sendBeacon also support fetch keep alive, my understanding is that it is still considered to be expermenital.

user-agent is a SHOULD requirement not a MUST requirement: https://github.com/open-telemetry/opentelemetry-specification/blob/ce2c5946faa9c961ea01d89c7aed83621865b67b/specification/protocol/exporter.md#user-agent

Further discussion: #3845

regardless I think the refactor should be done separately from this PR as it is a very significant change

dyladan · 2023-05-31T15:08:06Z

experimental/packages/otlp-exporter-base/src/platform/browser/util.ts

- * @param onSuccess
- * @param onError
- */
-export function sendWithBeacon(


This cannot be removed without a serious discussion and significant user benefit

I agree with @dyladan during page unloading sendBeacon and fetch (with keepalive) are the only options for getting data off the browser. And fetch with keep-alive is not always supported.

I checked the compatibility of these options. ~5% of the browsers support sendBeacon but not keepalive.
Maybe we should implement with fetch and use sendBeacon as a fallback?

Further discussion: #3845

llc1123 · 2023-05-31T15:17:37Z

I'd like to see some duplication reduction and the beacon sending should not be removed in this PR.

I also think this PR is quite large and each changelog entry should have its own PR so that the changes can be reviewed independently.

There are 3 major changes, which are dependent each other.

The current implementation of merging headers/metadata is totally wrong, headers are applied in incorrect order.
To make the user-agent header work properly, the above one should be fixed.
sendBeacon will not be used anymore (it doesn't support HTTP headers) and should be removed.

llc1123 · 2023-05-31T16:00:14Z

If this PR is too large, I would like to split it into following PRs:

Fix the applying order of config and env vars.
- Fix other env vars related issues.
Replace xhr and sendBeacon with fetch (keepalive=true).
Normalize the http header to lower cases.
Add the user-agent header to base packages.

MSNev

Do not remove sendBeacon support

llc1123 · 2023-05-31T16:48:28Z

Further discussion about the use of sendBeacon: #3845

dyladan · 2023-05-31T16:53:09Z

If this PR is too large, I would like to split it into following PRs:

Fix the applying order of config and env vars.

Fix other env vars related issues.

Replace xhr and sendBeacon with fetch (keepalive=true).

Normalize the http header to lower cases.

Add the user-agent header to base packages.

I would support that

dyladan · 2023-07-26T13:40:42Z

Any update on this?

llc1123 · 2023-07-26T13:45:08Z

Any update on this?

It's blocked by #3845 I think.

pichlermarc · 2023-12-12T12:23:06Z

Closing this for now - the changes proposed by PR were very controversial around removing sendBeacon.
I'm about to open a PR which proposes more modular exporters, I'll start with node.js will take on browser exporters in a follow-up, which will include user-agent headers when XHR is used for sending.

llc1123 force-pushed the fix/otlp-exporter-user-agent branch 2 times, most recently from 8111f47 to 3c01ac3 Compare May 17, 2023 09:02

llc1123 force-pushed the fix/otlp-exporter-user-agent branch 8 times, most recently from 0118b41 to f87d8d1 Compare May 18, 2023 04:32

llc1123 marked this pull request as ready for review May 18, 2023 04:40

llc1123 requested a review from a team May 18, 2023 04:40

llc1123 changed the title ~~[otlp-exporter] support user-agent headers in web exporters~~ feat(otlp-exporter): support user-agent headers in web exporters May 18, 2023

llc1123 mentioned this pull request May 18, 2023

[otlp-exporter] web exporters should set User-Agent header upon requests. #3808

Closed

llc1123 added 11 commits May 23, 2023 01:47

Revert "feat(otlp-metric-exporters): Add User-Agent header to OTLP me…

75c0391

…tric exporters (open-telemetry#3806)" This reverts commit 6a876a0.

Revert "feat(otlp-trace-exporters): Add User-Agent header to OTLP tra…

8592a08

…ce exporters (open-telemetry#3790)" This reverts commit 758c7af.

feat(otlp-exporter): add user-agent header to otlp exporters and re…

0fca00e

…move `sendBeacon` which does not support custom headers

fix(otlp-http-exporter): remove test cases of sendBeacon

786797a

feat(otlp-grpc-exporter): set metadata in correct order

5230b05

feat(otlp-http-exporter): set headers in correct order

bc67f46

feat(otlp-proto-exporter): set headers in correct order

8f77cd0

fix(otlp-exporter): normalize http headers to lower case

96009bd

chore: changelog

b9abac9

fix(otlp-grpc-exporter): env vars should not override configs from co…

040ac51

…nstructor parameter

chore(otlp-exporter): add user-agent header/metadata tests

361c1ac

llc1123 force-pushed the fix/otlp-exporter-user-agent branch from f87d8d1 to 361c1ac Compare May 22, 2023 17:52

Merge branch 'main' into fix/otlp-exporter-user-agent

828e89f

JamieDanielson reviewed May 23, 2023

View reviewed changes

experimental/packages/exporter-trace-otlp-grpc/test/OTLPTraceExporter.test.ts Outdated Show resolved Hide resolved

JamieDanielson reviewed May 23, 2023

View reviewed changes

experimental/packages/exporter-trace-otlp-grpc/test/OTLPTraceExporter.test.ts Outdated Show resolved Hide resolved

experimental/packages/exporter-trace-otlp-grpc/test/OTLPTraceExporter.test.ts Show resolved Hide resolved

llc1123 added 4 commits May 25, 2023 15:01

Merge branch 'main' into fix/otlp-exporter-user-agent

ea5632b

Merge branch 'main' into fix/otlp-exporter-user-agent

c7fada3

fix: refine tests

a0bbdce

Merge branch 'main' into fix/otlp-exporter-user-agent

f3f0ffd

llc1123 requested a review from JamieDanielson May 29, 2023 09:05

Merge branch 'main' into fix/otlp-exporter-user-agent

a0f8640

dyladan requested changes May 31, 2023

View reviewed changes

MSNev requested changes May 31, 2023

View reviewed changes

llc1123 mentioned this pull request Jun 3, 2023

REQUEST: New membership for llc1123 open-telemetry/community#1525

Closed

6 tasks

pichlermarc closed this Dec 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(otlp-exporter): support `user-agent` headers in web exporters #3811

feat(otlp-exporter): support `user-agent` headers in web exporters #3811

llc1123 commented May 17, 2023 •

edited

Loading

codecov bot commented May 17, 2023 •

edited

Loading

JamieDanielson left a comment

dyladan left a comment

dyladan May 31, 2023

llc1123 May 31, 2023

llc1123 May 31, 2023

dyladan May 31, 2023

llc1123 May 31, 2023

dyladan May 31, 2023

llc1123 May 31, 2023

llc1123 May 31, 2023

MSNev May 31, 2023

dyladan May 31, 2023

llc1123 May 31, 2023

dyladan May 31, 2023

dyladan May 31, 2023

MSNev May 31, 2023

llc1123 May 31, 2023

llc1123 May 31, 2023

llc1123 commented May 31, 2023

llc1123 commented May 31, 2023 •

edited

Loading

MSNev left a comment

llc1123 commented May 31, 2023

dyladan commented May 31, 2023

dyladan commented Jul 26, 2023

llc1123 commented Jul 26, 2023

pichlermarc commented Dec 12, 2023

feat(otlp-exporter): support user-agent headers in web exporters #3811

feat(otlp-exporter): support user-agent headers in web exporters #3811

Conversation

llc1123 commented May 17, 2023 • edited Loading

Which problem is this PR solving?

Short description of the changes

Type of change

How Has This Been Tested?

Checklist:

codecov bot commented May 17, 2023 • edited Loading

Codecov Report

JamieDanielson left a comment

Choose a reason for hiding this comment

dyladan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

llc1123 commented May 31, 2023

llc1123 commented May 31, 2023 • edited Loading

MSNev left a comment

Choose a reason for hiding this comment

llc1123 commented May 31, 2023

dyladan commented May 31, 2023

dyladan commented Jul 26, 2023

llc1123 commented Jul 26, 2023

pichlermarc commented Dec 12, 2023

feat(otlp-exporter): support `user-agent` headers in web exporters #3811

feat(otlp-exporter): support `user-agent` headers in web exporters #3811

llc1123 commented May 17, 2023 •

edited

Loading

codecov bot commented May 17, 2023 •

edited

Loading

llc1123 commented May 31, 2023 •

edited

Loading