Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make OTLP exporter memory mode API public #6469

Merged
merged 1 commit into from
May 30, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -1,2 +1,19 @@
Comparing source compatibility of against
No changes.
*** MODIFIED CLASS: PUBLIC FINAL io.opentelemetry.exporter.otlp.http.logs.OtlpHttpLogRecordExporterBuilder (not serializable)
=== CLASS FILE FORMAT VERSION: 52.0 <- 52.0
*** MODIFIED METHOD: PUBLIC (<- PACKAGE_PROTECTED) io.opentelemetry.exporter.otlp.http.logs.OtlpHttpLogRecordExporterBuilder setMemoryMode(io.opentelemetry.sdk.common.export.MemoryMode)
*** MODIFIED CLASS: PUBLIC FINAL io.opentelemetry.exporter.otlp.http.metrics.OtlpHttpMetricExporterBuilder (not serializable)
=== CLASS FILE FORMAT VERSION: 52.0 <- 52.0
*** MODIFIED METHOD: PUBLIC (<- PACKAGE_PROTECTED) io.opentelemetry.exporter.otlp.http.metrics.OtlpHttpMetricExporterBuilder setMemoryMode(io.opentelemetry.sdk.common.export.MemoryMode)
*** MODIFIED CLASS: PUBLIC FINAL io.opentelemetry.exporter.otlp.http.trace.OtlpHttpSpanExporterBuilder (not serializable)
=== CLASS FILE FORMAT VERSION: 52.0 <- 52.0
*** MODIFIED METHOD: PUBLIC (<- PACKAGE_PROTECTED) io.opentelemetry.exporter.otlp.http.trace.OtlpHttpSpanExporterBuilder setMemoryMode(io.opentelemetry.sdk.common.export.MemoryMode)
*** MODIFIED CLASS: PUBLIC FINAL io.opentelemetry.exporter.otlp.logs.OtlpGrpcLogRecordExporterBuilder (not serializable)
=== CLASS FILE FORMAT VERSION: 52.0 <- 52.0
*** MODIFIED METHOD: PUBLIC (<- PACKAGE_PROTECTED) io.opentelemetry.exporter.otlp.logs.OtlpGrpcLogRecordExporterBuilder setMemoryMode(io.opentelemetry.sdk.common.export.MemoryMode)
*** MODIFIED CLASS: PUBLIC FINAL io.opentelemetry.exporter.otlp.metrics.OtlpGrpcMetricExporterBuilder (not serializable)
=== CLASS FILE FORMAT VERSION: 52.0 <- 52.0
*** MODIFIED METHOD: PUBLIC (<- PACKAGE_PROTECTED) io.opentelemetry.exporter.otlp.metrics.OtlpGrpcMetricExporterBuilder setMemoryMode(io.opentelemetry.sdk.common.export.MemoryMode)
*** MODIFIED CLASS: PUBLIC FINAL io.opentelemetry.exporter.otlp.trace.OtlpGrpcSpanExporterBuilder (not serializable)
=== CLASS FILE FORMAT VERSION: 52.0 <- 52.0
*** MODIFIED METHOD: PUBLIC (<- PACKAGE_PROTECTED) io.opentelemetry.exporter.otlp.trace.OtlpGrpcSpanExporterBuilder setMemoryMode(io.opentelemetry.sdk.common.export.MemoryMode)
Original file line number Diff line number Diff line change
Expand Up @@ -210,8 +210,13 @@ public OtlpHttpLogRecordExporterBuilder setMeterProvider(
return this;
}

/** Set the {@link MemoryMode}. */
OtlpHttpLogRecordExporterBuilder setMemoryMode(MemoryMode memoryMode) {
/**
* Set the {@link MemoryMode}. If unset, defaults to {@link #DEFAULT_MEMORY_MODE}.
*
* <p>>When memory mode is {@link MemoryMode#REUSABLE_DATA}, serialization is optimized to reduce
* memory allocation.
*/
public OtlpHttpLogRecordExporterBuilder setMemoryMode(MemoryMode memoryMode) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do I understand correctly that this is part of public api of a stable module and once we have added this method we can't easily get rid of it? If so then perhaps it would be better to not tie this to implementation details like immutable or reusable data, but rather something abstract like minimize allocations and maximize throughput (I guess this would only make sense if the one that allocates more has a bit better performance). That way the behavior of this method could more easily accommodate future changes which could, for example, include deleting one of the memory modes.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do I understand correctly that this is part of public api of a stable module and once we have added this method we can't easily get rid of it?

We can't get rid of it without a major version bump, which we don't plan on doing anytime soon.

perhaps it would be better to not tie this to implementation details like immutable or reusable data, but rather something abstract like minimize allocations and maximize throughput (I guess this would only make sense if the one that allocates more has a bit better performance). That way the behavior of this method could more easily accommodate future changes which could, for example, include deleting one of the memory modes.

The MemoryMode enum is also part of the stable API at this point so we can't delete the memory modes. In hindsight it might have been preferable to choose a name like MemoryMode.LOW_ALLOCATION instead MemoryMode.REUSABLE_DATA. LOW_ALLOCATION describes the intended outcome (more user facing) where REUSABLE_DATA describes what is happening under the covers. The flip side of this is that REUSABLE_DATA communicates some important information to MetricReader / MetricExporter implementations: we're going to reuse MetricData classes so they won't function right after the CompletableResultCode from MetricExporter.export() resolves. So there's benefits to naming for the outcome and also how the outcome is accomplished.

REUSABLE_DATA doesn't describes what's happening with the serializers as well as it describes what's happening with the metrics SDK where it was originally introduced, but its still somewhat accurate. One thing that sticks out is that with the metrics SDK, implementers of MetricReader / MetricExporter have to be aware of the semantics of REUSABLE_DATA. With serializers, there's no impact to user semantics. The only indication a user can see that something has changed is a shift in the CPU / memory behavior.

Overall, I think its preferable to have one memory mode configuration concept, even if the words we chose for that concept (IMMUTABLE_DATA and REUSABLE_DATA) don't perfectly describe all the places we make that configurable.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The flip side of this is that REUSABLE_DATA communicates some important information to MetricReader / MetricExporter implementations: we're going to reuse MetricData classes so they won't function right after the CompletableResultCode from MetricExporter.export() resolves

👍

requireNonNull(memoryMode, "memoryMode");
this.memoryMode = memoryMode;
return this;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,10 +19,12 @@
import io.opentelemetry.sdk.common.export.ProxyOptions;
import io.opentelemetry.sdk.common.export.RetryPolicy;
import io.opentelemetry.sdk.metrics.InstrumentType;
import io.opentelemetry.sdk.metrics.data.MetricData;
import io.opentelemetry.sdk.metrics.export.AggregationTemporalitySelector;
import io.opentelemetry.sdk.metrics.export.DefaultAggregationSelector;
import io.opentelemetry.sdk.metrics.export.MetricExporter;
import java.time.Duration;
import java.util.Collection;
import java.util.Map;
import java.util.concurrent.TimeUnit;
import java.util.function.Supplier;
Expand Down Expand Up @@ -231,8 +233,18 @@ public OtlpHttpMetricExporterBuilder setProxyOptions(ProxyOptions proxyOptions)
return this;
}

/** Set the {@link MemoryMode}. */
OtlpHttpMetricExporterBuilder setMemoryMode(MemoryMode memoryMode) {
/**
* Set the {@link MemoryMode}. If unset, defaults to {@link #DEFAULT_MEMORY_MODE}.
*
* <p>>When memory mode is {@link MemoryMode#REUSABLE_DATA}, serialization is optimized to reduce
* memory allocation. Additionally, the value is used for {@link MetricExporter#getMemoryMode()},
* which sends a signal to the metrics SDK to reuse memory when possible. This is safe and
* desirable for most use cases, but should be used with caution of wrapping and delegating to the
* exporter. It is not safe for the wrapping exporter to hold onto references to {@link
* MetricData} batches since the same data structures will be reused in subsequent calls to {@link
* MetricExporter#export(Collection)}.
*/
public OtlpHttpMetricExporterBuilder setMemoryMode(MemoryMode memoryMode) {
requireNonNull(memoryMode, "memoryMode");
this.memoryMode = memoryMode;
return this;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -211,8 +211,13 @@ public OtlpHttpSpanExporterBuilder setMeterProvider(
return this;
}

/** Set the {@link MemoryMode}. */
OtlpHttpSpanExporterBuilder setMemoryMode(MemoryMode memoryMode) {
/**
* Set the {@link MemoryMode}. If unset, defaults to {@link #DEFAULT_MEMORY_MODE}.
*
* <p>>When memory mode is {@link MemoryMode#REUSABLE_DATA}, serialization is optimized to reduce
* memory allocation.
*/
public OtlpHttpSpanExporterBuilder setMemoryMode(MemoryMode memoryMode) {
requireNonNull(memoryMode, "memoryMode");
this.memoryMode = memoryMode;
return this;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,7 @@

import static io.opentelemetry.sdk.metrics.Aggregation.explicitBucketHistogram;

import io.opentelemetry.exporter.otlp.http.logs.OtlpHttpLogRecordExporterBuilder;
import io.opentelemetry.exporter.otlp.http.metrics.OtlpHttpMetricExporterBuilder;
import io.opentelemetry.exporter.otlp.http.trace.OtlpHttpSpanExporterBuilder;
import io.opentelemetry.exporter.otlp.logs.OtlpGrpcLogRecordExporterBuilder;
import io.opentelemetry.exporter.otlp.metrics.OtlpGrpcMetricExporterBuilder;
import io.opentelemetry.exporter.otlp.trace.OtlpGrpcSpanExporterBuilder;
import io.opentelemetry.exporter.internal.ExporterBuilderUtil;
import io.opentelemetry.sdk.autoconfigure.spi.ConfigProperties;
import io.opentelemetry.sdk.autoconfigure.spi.ConfigurationException;
import io.opentelemetry.sdk.common.export.MemoryMode;
Expand All @@ -26,8 +21,6 @@
import java.io.File;
import java.io.IOException;
import java.io.RandomAccessFile;
import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;
import java.net.MalformedURLException;
import java.net.URL;
import java.net.URLDecoder;
Expand Down Expand Up @@ -62,6 +55,7 @@ public static String getOtlpProtocol(String dataType, ConfigProperties config) {
}

/** Invoke the setters with the OTLP configuration for the {@code dataType}. */
@SuppressWarnings("TooManyParameters")
public static void configureOtlpExporterBuilder(
String dataType,
ConfigProperties config,
Expand All @@ -71,7 +65,8 @@ public static void configureOtlpExporterBuilder(
Consumer<Duration> setTimeout,
Consumer<byte[]> setTrustedCertificates,
BiConsumer<byte[], byte[]> setClientTls,
Consumer<RetryPolicy> setRetryPolicy) {
Consumer<RetryPolicy> setRetryPolicy,
Consumer<MemoryMode> setMemoryMode) {
String protocol = getOtlpProtocol(dataType, config);
boolean isHttpProtobuf = protocol.equals(PROTOCOL_HTTP_PROTOBUF);
URL endpoint =
Expand Down Expand Up @@ -161,6 +156,8 @@ public static void configureOtlpExporterBuilder(
if (retryEnabled) {
setRetryPolicy.accept(RetryPolicy.getDefault());
}

ExporterBuilderUtil.configureExporterMemoryMode(config, setMemoryMode);
}

/**
Expand Down Expand Up @@ -215,82 +212,6 @@ public static void configureOtlpHistogramDefaultAggregation(
}
}

/**
* Calls {@code #setMemoryMode} on the {@code Otlp{Protocol}{Signal}ExporterBuilder} with the
* {@code memoryMode}.
*/
public static void setMemoryModeOnOtlpExporterBuilder(Object builder, MemoryMode memoryMode) {
try {
// Metrics
if (builder instanceof OtlpGrpcMetricExporterBuilder) {
// Calling getDeclaredMethod causes all private methods to be read, which causes a
// ClassNotFoundException when running with the OkHttHttpProvider as the private
// setManagedChanel(io.grpc.ManagedChannel) is reached and io.grpc.ManagedChannel is not on
// the classpath. io.opentelemetry.exporter.otlp.metrics.OtlpGrpcMetricUtil provides a layer
// of indirection which avoids scanning the OtlpGrpcMetricExporterBuilder private methods.
Class<?> otlpGrpcMetricUtil =
Class.forName("io.opentelemetry.exporter.otlp.metrics.OtlpGrpcMetricUtil");
Method method =
otlpGrpcMetricUtil.getDeclaredMethod(
"setMemoryMode", OtlpGrpcMetricExporterBuilder.class, MemoryMode.class);
method.setAccessible(true);
method.invoke(null, builder, memoryMode);
} else if (builder instanceof OtlpHttpMetricExporterBuilder) {
Method method =
OtlpHttpMetricExporterBuilder.class.getDeclaredMethod(
"setMemoryMode", MemoryMode.class);
method.setAccessible(true);
method.invoke(builder, memoryMode);
} else if (builder instanceof OtlpGrpcSpanExporterBuilder) {
// Calling getDeclaredMethod causes all private methods to be read, which causes a
// ClassNotFoundException when running with the OkHttHttpProvider as the private
// setManagedChanel(io.grpc.ManagedChannel) is reached and io.grpc.ManagedChannel is not on
// the classpath. io.opentelemetry.exporter.otlp.trace.OtlpGrpcSpanUtil provides a layer
// of indirection which avoids scanning the OtlpGrpcSpanExporterBuilder private methods.
Class<?> otlpGrpcMetricUtil =
Class.forName("io.opentelemetry.exporter.otlp.trace.OtlpGrpcSpanUtil");
Method method =
otlpGrpcMetricUtil.getDeclaredMethod(
"setMemoryMode", OtlpGrpcSpanExporterBuilder.class, MemoryMode.class);
method.setAccessible(true);
method.invoke(null, builder, memoryMode);
} else if (builder instanceof OtlpHttpSpanExporterBuilder) {
Method method =
OtlpHttpSpanExporterBuilder.class.getDeclaredMethod("setMemoryMode", MemoryMode.class);
method.setAccessible(true);
method.invoke(builder, memoryMode);
} else if (builder instanceof OtlpGrpcLogRecordExporterBuilder) {
// Calling getDeclaredMethod causes all private methods to be read, which causes a
// ClassNotFoundException when running with the OkHttHttpProvider as the private
// setManagedChanel(io.grpc.ManagedChannel) is reached and io.grpc.ManagedChannel is not on
// the classpath. io.opentelemetry.exporter.otlp.logs.OtlpGrpcLogUtil provides a layer
// of indirection which avoids scanning the OtlpGrpcLogRecordExporterBuilder private
// methods.
Class<?> otlpGrpcMetricUtil =
Class.forName("io.opentelemetry.exporter.otlp.logs.OtlpGrpcLogUtil");
Method method =
otlpGrpcMetricUtil.getDeclaredMethod(
"setMemoryMode", OtlpGrpcLogRecordExporterBuilder.class, MemoryMode.class);
method.setAccessible(true);
method.invoke(null, builder, memoryMode);
} else if (builder instanceof OtlpHttpLogRecordExporterBuilder) {
Method method =
OtlpHttpLogRecordExporterBuilder.class.getDeclaredMethod(
"setMemoryMode", MemoryMode.class);
method.setAccessible(true);
method.invoke(builder, memoryMode);
} else {
throw new IllegalArgumentException(
"Cannot set memory mode. Unrecognized OTLP exporter builder");
}
} catch (NoSuchMethodException
| InvocationTargetException
| IllegalAccessException
| ClassNotFoundException e) {
throw new IllegalStateException("Error calling setMemoryMode.", e);
}
}

private static URL createUrl(URL context, String spec) {
try {
return new URL(context, spec);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,6 @@
import static io.opentelemetry.exporter.otlp.internal.OtlpConfigUtil.PROTOCOL_HTTP_PROTOBUF;

import io.opentelemetry.api.metrics.MeterProvider;
import io.opentelemetry.exporter.internal.ExporterBuilderUtil;
import io.opentelemetry.exporter.otlp.http.logs.OtlpHttpLogRecordExporter;
import io.opentelemetry.exporter.otlp.http.logs.OtlpHttpLogRecordExporterBuilder;
import io.opentelemetry.exporter.otlp.logs.OtlpGrpcLogRecordExporter;
Expand Down Expand Up @@ -52,11 +51,9 @@ public LogRecordExporter createExporter(ConfigProperties config) {
builder::setTimeout,
builder::setTrustedCertificates,
builder::setClientTls,
builder::setRetryPolicy);
builder::setRetryPolicy,
builder::setMemoryMode);
builder.setMeterProvider(meterProviderRef::get);
ExporterBuilderUtil.configureExporterMemoryMode(
config,
memoryMode -> OtlpConfigUtil.setMemoryModeOnOtlpExporterBuilder(builder, memoryMode));

return builder.build();
} else if (protocol.equals(PROTOCOL_GRPC)) {
Expand All @@ -71,11 +68,9 @@ public LogRecordExporter createExporter(ConfigProperties config) {
builder::setTimeout,
builder::setTrustedCertificates,
builder::setClientTls,
builder::setRetryPolicy);
builder::setRetryPolicy,
builder::setMemoryMode);
builder.setMeterProvider(meterProviderRef::get);
ExporterBuilderUtil.configureExporterMemoryMode(
config,
memoryMode -> OtlpConfigUtil.setMemoryModeOnOtlpExporterBuilder(builder, memoryMode));

return builder.build();
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@
import static io.opentelemetry.exporter.otlp.internal.OtlpConfigUtil.PROTOCOL_GRPC;
import static io.opentelemetry.exporter.otlp.internal.OtlpConfigUtil.PROTOCOL_HTTP_PROTOBUF;

import io.opentelemetry.exporter.internal.ExporterBuilderUtil;
import io.opentelemetry.exporter.otlp.http.metrics.OtlpHttpMetricExporter;
import io.opentelemetry.exporter.otlp.http.metrics.OtlpHttpMetricExporterBuilder;
import io.opentelemetry.exporter.otlp.metrics.OtlpGrpcMetricExporter;
Expand Down Expand Up @@ -44,14 +43,12 @@ public MetricExporter createExporter(ConfigProperties config) {
builder::setTimeout,
builder::setTrustedCertificates,
builder::setClientTls,
builder::setRetryPolicy);
builder::setRetryPolicy,
builder::setMemoryMode);
OtlpConfigUtil.configureOtlpAggregationTemporality(
config, builder::setAggregationTemporalitySelector);
OtlpConfigUtil.configureOtlpHistogramDefaultAggregation(
config, builder::setDefaultAggregationSelector);
ExporterBuilderUtil.configureExporterMemoryMode(
config,
memoryMode -> OtlpConfigUtil.setMemoryModeOnOtlpExporterBuilder(builder, memoryMode));

return builder.build();
} else if (protocol.equals(PROTOCOL_GRPC)) {
Expand All @@ -66,14 +63,12 @@ public MetricExporter createExporter(ConfigProperties config) {
builder::setTimeout,
builder::setTrustedCertificates,
builder::setClientTls,
builder::setRetryPolicy);
builder::setRetryPolicy,
builder::setMemoryMode);
OtlpConfigUtil.configureOtlpAggregationTemporality(
config, builder::setAggregationTemporalitySelector);
OtlpConfigUtil.configureOtlpHistogramDefaultAggregation(
config, builder::setDefaultAggregationSelector);
ExporterBuilderUtil.configureExporterMemoryMode(
config,
memoryMode -> OtlpConfigUtil.setMemoryModeOnOtlpExporterBuilder(builder, memoryMode));

return builder.build();
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,6 @@
import static io.opentelemetry.exporter.otlp.internal.OtlpConfigUtil.PROTOCOL_HTTP_PROTOBUF;

import io.opentelemetry.api.metrics.MeterProvider;
import io.opentelemetry.exporter.internal.ExporterBuilderUtil;
import io.opentelemetry.exporter.otlp.http.trace.OtlpHttpSpanExporter;
import io.opentelemetry.exporter.otlp.http.trace.OtlpHttpSpanExporterBuilder;
import io.opentelemetry.exporter.otlp.trace.OtlpGrpcSpanExporter;
Expand Down Expand Up @@ -51,11 +50,10 @@ public SpanExporter createExporter(ConfigProperties config) {
builder::setTimeout,
builder::setTrustedCertificates,
builder::setClientTls,
builder::setRetryPolicy);
builder::setRetryPolicy,
builder::setMemoryMode);
builder.setMeterProvider(meterProviderRef::get);
ExporterBuilderUtil.configureExporterMemoryMode(
config,
memoryMode -> OtlpConfigUtil.setMemoryModeOnOtlpExporterBuilder(builder, memoryMode));

return builder.build();
} else if (protocol.equals(PROTOCOL_GRPC)) {
OtlpGrpcSpanExporterBuilder builder = grpcBuilder();
Expand All @@ -69,11 +67,9 @@ public SpanExporter createExporter(ConfigProperties config) {
builder::setTimeout,
builder::setTrustedCertificates,
builder::setClientTls,
builder::setRetryPolicy);
builder::setRetryPolicy,
builder::setMemoryMode);
builder.setMeterProvider(meterProviderRef::get);
ExporterBuilderUtil.configureExporterMemoryMode(
config,
memoryMode -> OtlpConfigUtil.setMemoryModeOnOtlpExporterBuilder(builder, memoryMode));

return builder.build();
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -243,8 +243,13 @@ public OtlpGrpcLogRecordExporterBuilder setMeterProvider(
return this;
}

/** Set the {@link MemoryMode}. */
OtlpGrpcLogRecordExporterBuilder setMemoryMode(MemoryMode memoryMode) {
/**
* Set the {@link MemoryMode}. If unset, defaults to {@link #DEFAULT_MEMORY_MODE}.
*
* <p>>When memory mode is {@link MemoryMode#REUSABLE_DATA}, serialization is optimized to reduce
* memory allocation.
*/
public OtlpGrpcLogRecordExporterBuilder setMemoryMode(MemoryMode memoryMode) {
requireNonNull(memoryMode, "memoryMode");
this.memoryMode = memoryMode;
return this;
Expand Down

This file was deleted.

Loading
Loading