OTLP ingestion with gzip compression is broken

[glucaci]

APM Server version (apm-server version): 8.9.1

Description of the problem including expected versus actual behavior:

After upgrading the elastic cluster from 8.9.0 to 8.9.1 the opentelemetry-collector is not working anymore with gzip compression.
gzip is the default compression used by the opentelemetry-collector

As workaround the compression can be disabled in opentelemetry-collector configuration

exporters:
  otlp/elastic:
    endpoint: ${OTLP_ELASTIC_ENDPOINT}
    compression: none

Steps to reproduce:

1. Setup Elastic Cluster with version 8.9.1
2. Setup opentelemetry-collector with default configuration

Provide logs (if relevant):

2023-08-29T10:19:08.609Z	error	exporterhelper/queued_retry.go:391	Exporting failed. The error is not retryable. Dropping data.	{"kind": "exporter", "data_type": "traces", "name": "otlp/elastic", "error": "Permanent error: rpc error: code = Unimplemented desc = grpc: Decompressor is not installed for grpc-encoding \"gzip\"", "dropped_items": 1}
go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
	go.opentelemetry.io/collector/[email protected]/exporterhelper/queued_retry.go:391
go.opentelemetry.io/collector/exporter/exporterhelper.(*tracesExporterWithObservability).send
	go.opentelemetry.io/collector/[email protected]/exporterhelper/traces.go:126
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1
	go.opentelemetry.io/collector/[email protected]/exporterhelper/queued_retry.go:195
go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func1
	go.opentelemetry.io/collector/[email protected]/exporterhelper/internal/bounded_memory_queue.go:47

[axw]

@glucaci sorry you got caught by this, and thanks for documenting the workaround. This bug was fixed last week (#11434) and backported to 8.9 -- it will be in 8.9.2. As part of the fix we've improved our testing to avoid a regression. Closing as there's nothing to do but wait (we're expecting 8.9.2 in the next couple of weeks).