Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Graduate the new googlecloud exporter to beta #9471

Merged
merged 1 commit into from
May 2, 2022
Merged

Graduate the new googlecloud exporter to beta #9471

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@
- `kafkametricsreceiver`: instrumentation name updated from `otelcol/kafkametrics` to `otelcol/kafkametricsreceiver` (#9406)
- `kubeletstatsreceiver`: instrumentation name updated from `kubeletstats` to `otelcol/kubeletstatsreceiver` (#9400)
- `datadogexporter`: Remove `GetHostTags` method from `TagsConfig` struct (#9423)
- `googlecloudexporter`: Graduate the `exporter.googlecloud.OTLPDirect` feature-gate to Beta. This includes changes to the configuration structure, and many changes to default behavior. (#9471)

### 🚩 Deprecations 🚩

Expand Down
124 changes: 62 additions & 62 deletions exporter/googlecloudexporter/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -124,12 +124,7 @@ These instructions are to get you up and running quickly with the GCP exporter i
The following configuration options are supported:

- `project` (optional): GCP project identifier.
- `endpoint` (optional): Endpoint where data is going to be sent to.
- `user_agent` (optional): Override the user agent string sent on requests to Cloud Monitoring (currently only applies to metrics). Specify `{{version}}` to include the application version number. Defaults to `opentelemetry-collector-contrib {{version}}`.
- `use_insecure` (optional): If true. use gRPC as their communication transport. Only has effect if Endpoint is not "".
- `timeout` (optional): Timeout for all API calls. If not set, defaults to 12 seconds.
- `resource_mappings` (optional): ResourceMapping defines mapping of resources from source (OpenCensus) to target (Google Cloud).
- `label_mappings` (optional): Optional flag signals whether we can proceed with transformation if a label is missing in the resource.
- `retry_on_failure` (optional): Configuration for how to handle retries when sending data to Google Cloud fails.
- `enabled` (default = true)
- `initial_interval` (default = 5s): Time to wait after the first failure before retrying; ignored if `enabled` is `false`
Expand All @@ -147,8 +142,22 @@ Note: These `retry_on_failure` and `sending_queue` are provided (and documented)

Additional configuration for the metric exporter:

- `metric.prefix` (optional): MetricPrefix overrides the prefix / namespace of the Google Cloud metric type identifier. If not set, defaults to "custom.googleapis.com/opencensus/"
- `metric.skip_create_descriptor` (optional): Whether to skip creating the metric descriptor.
- `metric.endpoint` (optional): Endpoint where metric data is going to be sent to. Replaces `endpoint`.
- `metric.use_insecure` (optional): If true, use gRPC as their communication transport. Only has effect if Endpoint is not "".
- `metric.known_domains` (optional): If a metric belongs to one of these domains it does not get a prefix.
- `metric.skip_create_descriptor` (optional): If set to true, do not send metric descriptors to GCM.
- `metric.instrumentation_library_labels` (optional): If true, set the instrumentation_source and instrumentation_version labels. Defaults to true.
- `metric.create_service_timeseries` (optional): If true, this will send all timeseries using `CreateServiceTimeSeries`. Implicitly, this sets `skip_create_descriptor` to true.
- `metric.create_metric_descriptor_buffer_size` (optional): Buffer size for the channel which asynchronously calls CreateMetricDescriptor. Default is 10.
- `metric.service_resource_labels` (optional): If true, the exporter will copy OTel's service.name, service.namespace, and service.instance.id resource attributes into the GCM timeseries metric labels. Default is true.
- `metric.resource_filters` (optional): If provided, resource attributes matching any filter will be included in metric labels. Defaults to empty, which won't include any additional resource labels.
- `prefix`: Match resource keys by prefix

Additional configuration added for the trace exporter:

- `trace.endpoint` (optional): Endpoint where trace data is going to be sent to. Replaces `endpoint`.
- `trace.use_insecure` (optional): If true. use gRPC as their communication transport. Only has effect if Endpoint is not "". Replaces `use_insecure`.
- `trace.attribute_mappings` (optional): AttributeMappings determines how to map from OpenTelemetry attribute keys to Google Cloud Trace keys. By default, it changes http and service keys so that they appear more prominently in the UI.

Example:

Expand All @@ -161,29 +170,31 @@ exporters:
retry_on_failure:
enabled: false
project: my-project
endpoint: test-endpoint
user_agent: my-collector {{version}}
use_insecure: true
timeout: 12s

resource_mappings:
- source_type: source.resource1
target_type: target-resource1
label_mappings:
- source_key: contrib.opencensus.io/exporter/googlecloud/project_id
target_key: project_id
optional: true
- source_key: source.label1
target_key: target_label_1
trace:
endpoint: cloudtrace.googleapis.com
use_insecure: false
attribute_mappings:
- key: service.name
replacement: g.co/gae/app/module
metric:
endpoint: monitoring.googleapis.com
use_insecure: false
prefix: workload.googleapis.com
skip_create_descriptor: false
known_domains:
- mycustomdomain.googleapis.com
instrumentation_library_labels: true
service_resource_labels: true
create_service_timeseries: false
create_metric_descriptor_buffer_size: 10
resource_filters:
- prefix: k8s.*

sending_queue:
enabled: true
num_consumers: 2
queue_size: 50

metric:
prefix: prefix
skip_create_descriptor: true
```

Beyond standard YAML configuration as outlined in the sections that follow,
Expand All @@ -209,48 +220,37 @@ optimal network usage and avoiding memory overruns. You may also want to run an

See the [Collector feature gates](https://github.com/open-telemetry/opentelemetry-collector/blob/main/service/featuregate/README.md#collector-feature-gates) for an overview of feature gates in the collector.

**ALPHA**: `exporter.googlecloud.OTLPDirect`

When enabled via `--feature-gates=exporter.googlecloud.OTLPDirect`, the googlecloud exporter translates pdata directly to google cloud monitoring's types, rather than first translating to opencensus. See the [Breaking Changes documentation](https://github.com/GoogleCloudPlatform/opentelemetry-operations-go/blob/main/exporter/collector/breaking-changes.md#breaking-changes-vs-old-googlecloud-exporter) for breaking changes that will occur as a result of enabling this feature.

Additional configuration added for the metric exporter:

- `metric.endpoint` (optional): Endpoint where metric data is going to be sent to. Replaces `endpoint`.
- `metric.use_insecure` (optional): If true. use gRPC as their communication transport. Only has effect if Endpoint is not "". Replaces `use_insecure`.
- `metric.known_domains` (optional): If a metric belongs to one of these domains it does not get a prefix.
- `metric.instrumentation_library_labels` (optional): If true, set the instrumentation_source and instrumentation_version labels. Defaults to true.
- `metric.create_service_timeseries` (optional): If true, this will send all timeseries using `CreateServiceTimeSeries`. Implicitly, this sets `SkipMetricDescriptor` to true.
- `metric.create_metric_descriptor_buffer_size` (optional): Buffer size for the channel which asynchronously calls CreateMetricDescriptor. Default is 10.
- `metric.service_resource_labels` (optional): If true, the exporter will copy OTel's service.name, service.namespace, and service.instance.id resource attributes into the GCM timeseries metric labels. Default is true.
- `metric.resource_filters` (optional): If provided, resource attributes matching any filter will be included in metric labels. Defaults to empty, which won't include any additional resource labels.
- `prefix`: Match resource keys by prefix

Additional configuration added for the trace exporter:

- `trace.endpoint` (optional): Endpoint where trace data is going to be sent to. Replaces `endpoint`.
- `trace.use_insecure` (optional): If true. use gRPC as their communication transport. Only has effect if Endpoint is not "". Replaces `use_insecure`.

Removed configuration:

- `endpoint`: replaced by `trace.endpoint` and `metric.endpoint`
- `use_insecure`: replaced by `trace.use_insecure` and `metric.use_insecure`
- `resource_mappings`: replacement is still under development
**BETA**: `exporter.googlecloud.OTLPDirect`

Changes to defaults:
The `exporter.googlecloud.OTLPDirect` is enabled by default starting in v0.50.0, and can be disabled via `--feature-gates=-exporter.googlecloud.OTLPDirect`. The new googlecloud exporter translates pdata directly to google cloud monitoring's types, rather than first translating to opencensus. See the [Breaking Changes documentation](https://github.com/GoogleCloudPlatform/opentelemetry-operations-go/blob/main/exporter/collector/breaking-changes.md#breaking-changes-vs-old-googlecloud-exporter) for breaking changes that will occur as a result of this feature.

- `prefix`: default is now `workload.googleapis.com`
If you are broken by changes described there, or have encountered an issue with the new implementation, please open an issue [here](https://github.com/GoogleCloudPlatform/opentelemetry-operations-go/issues/new)

Additional Behavior:
If you disable the feature-gate, you can continue to set the legacy configuration options:

- Metrics are now batched in the exporter to handle edge cases around summar metrics. The batch processor is no longer needed in metrics processors.
- `project` (optional): GCP project identifier.
- `endpoint` (optional): Endpoint where data is going to be sent to.
- `user_agent` (optional): Override the user agent string sent on requests to Cloud Monitoring (currently only applies to metrics). Specify `{{version}}` to include the application version number. Defaults to `opentelemetry-collector-contrib {{version}}`.
- `use_insecure` (optional): If true. use gRPC as their communication transport. Only has effect if Endpoint is not "".
- `timeout` (optional): Timeout for all API calls. If not set, defaults to 12 seconds.
- `resource_mappings` (optional): ResourceMapping defines mapping of resources from source (OpenCensus) to target (Google Cloud).
- `label_mappings` (optional): Optional flag signals whether we can proceed with transformation if a label is missing in the resource.
- `retry_on_failure` (optional): Configuration for how to handle retries when sending data to Google Cloud fails.
- `enabled` (default = true)
- `initial_interval` (default = 5s): Time to wait after the first failure before retrying; ignored if `enabled` is `false`
- `max_interval` (default = 30s): Is the upper bound on backoff; ignored if `enabled` is `false`
- `max_elapsed_time` (default = 120s): Is the maximum amount of time spent trying to send a batch; ignored if `enabled` is `false`
- `sending_queue` (optional): Configuration for how to buffer traces before sending.
- `enabled` (default = true)
- `num_consumers` (default = 10): Number of consumers that dequeue batches; ignored if `enabled` is `false`
- `queue_size` (default = 5000): Maximum number of batches kept in memory before data; ignored if `enabled` is `false`;
User should calculate this as `num_seconds * requests_per_second` where:
- `num_seconds` is the number of seconds to buffer in case of a backend outage
- `requests_per_second` is the average number of requests per seconds.

## Deprecatations
Note: These `retry_on_failure` and `sending_queue` are provided (and documented) by the [Exporter Helper](https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/exporterhelper#configuration)

The previous trace configuration (v0.21.0) has been deprecated in favor of the common configuration options available in OpenTelemetry. These will cause a failure to start
and should be migrated:
Additional configuration for the metric exporter:

- `trace.bundle_delay_threshold` (optional): Use `batch` processor instead ([docs](https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor/batchprocessor)).
- `trace.bundle_count_threshold` (optional): Use `batch` processor instead ([docs](https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor/batchprocessor)).
- `trace.bundle_byte_threshold` (optional): Use `memorylimiter` processor instead ([docs](https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor/memorylimiterprocessor))
- `trace.bundle_byte_limit` (optional): Use `memorylimiter` processor instead ([docs](https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor/memorylimiterprocessor))
- `trace.buffer_max_bytes` (optional): Use `memorylimiter` processor instead ([docs](https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor/memorylimiterprocessor))
- `metric.prefix` (optional): MetricPrefix overrides the prefix / namespace of the Google Cloud metric type identifier. If not set, defaults to "custom.googleapis.com/opencensus/"
- `metric.skip_create_descriptor` (optional): Whether to skip creating the metric descriptor.
2 changes: 1 addition & 1 deletion exporter/googlecloudexporter/factory.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ func init() {
featuregate.GetRegistry().MustRegister(featuregate.Gate{
ID: pdataExporterFeatureGate,
Description: "When enabled, the googlecloud exporter translates pdata directly to google cloud monitoring's types, rather than first translating to opencensus.",
Enabled: false,
Enabled: true,
})
}

Expand Down