[exporter/awss3]: add marshaller for Sumo Logic Installed Collector format

[aboguszewski-sumo]

Component(s)

exporter/awss3

Is your feature request related to a problem? Please describe.

We would like to add a marshaller to the S3 exporter that will export the data in the format of deprecated Sumo Logic Installed Collector. An example use case is to send the data to S3 in order to send them to Sumo Logic later when needed.

Describe the solution you'd like

A reasonable solution would be to add an additional config option:

exporters:
  awss3:
    ## default = json
    marshaller: {json, sumologic_ic, ...}

It is already partially supported in the code (see paragraph "additional context").

Describe alternatives you've considered

An alternative would be to use this extension: #28686
However, it's still very early in the development phase and no code has been merged yet, so we can't use it now.

Additional context

Current implementation of the exporter allows to add other marshallers:

opentelemetry-collector-contrib/exporter/awss3exporter/marshaler.go

Line 26 in 8dc41a8

func NewMarshaler(mType MarshalerType, logger *zap.Logger) (marshaler, error) {

opentelemetry-collector-contrib/exporter/awss3exporter/config.go

Line 25 in 8dc41a8

MarshalerName MarshalerType `mapstructure:"marshaler"`

There is a PoC for logs here: https://github.com/SumoLogic/sumologic-otel-collector/blob/sumo_ic_marshaler/pkg/translator/sumoicmarshaler/sumo_ic_marshaler.go but it will undergo some changes before merging it to contrib repo (adding support for metrics and traces etc.).

[github-actions]

Pinging code owners:

• exporter/awss3: @atoulme @pdelewski

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[ledor473]

I was about to file something similar so I'll add my thoughts here.

Currently, the exporter/awss3 only supports OtlpJSON and there's no way to extend it aside from forking it. In comparison, the exporter/kafka is better structured to allow for extending it if you are building your own OTel Collector:

• It exports the Trace / Metrics / Logs marshaler interfaces so they can be implemented outside of this package:

  opentelemetry-collector-contrib/exporter/kafkaexporter/marshaler.go

  Lines 13 to 38 in 8dc41a8

  	// TracesMarshaler marshals traces into Message array.
  	type TracesMarshaler interface {
  	// Marshal serializes spans into sarama's ProducerMessages
  	Marshal(traces ptrace.Traces, topic string) ([]*sarama.ProducerMessage, error)
  	// Encoding returns encoding name
  	Encoding() string
  	}
  	// MetricsMarshaler marshals metrics into Message array
  	type MetricsMarshaler interface {
  	// Marshal serializes metrics into sarama's ProducerMessages
  	Marshal(metrics pmetric.Metrics, topic string) ([]*sarama.ProducerMessage, error)
  	// Encoding returns encoding name
  	Encoding() string
  	}
  	// LogsMarshaler marshals logs into Message array
  	type LogsMarshaler interface {
  	// Marshal serializes logs into sarama's ProducerMessages
  	Marshal(logs plog.Logs, topic string) ([]*sarama.ProducerMessage, error)
  	// Encoding returns encoding name
  	Encoding() string
  	}

• It provides a set of default implementation(s) (in the case of awss3 it could likely stay OtlpJSON only for now):

  opentelemetry-collector-contrib/exporter/kafkaexporter/factory.go

  Lines 72 to 77 in 8dc41a8

  	func NewFactory(options ...FactoryOption) exporter.Factory {
  	f := &kafkaExporterFactory{
  	tracesMarshalers: tracesMarshalers(),
  	metricsMarshalers: metricsMarshalers(),
  	logsMarshalers: logsMarshalers(),
  	}

• It provides a way to register other Marshaler that are custom-built:

  opentelemetry-collector-contrib/exporter/kafkaexporter/factory.go

  Lines 44 to 69 in 8dc41a8

  	// WithTracesMarshalers adds tracesMarshalers.
  	func WithTracesMarshalers(tracesMarshalers ...TracesMarshaler) FactoryOption {
  	return func(factory *kafkaExporterFactory) {
  	for _, marshaler := range tracesMarshalers {
  	factory.tracesMarshalers[marshaler.Encoding()] = marshaler
  	}
  	}
  	}
  	// WithMetricsMarshalers adds additional metric marshalers to the exporter factory.
  	func WithMetricsMarshalers(metricMarshalers ...MetricsMarshaler) FactoryOption {
  	return func(factory *kafkaExporterFactory) {
  	for _, marshaler := range metricMarshalers {
  	factory.metricsMarshalers[marshaler.Encoding()] = marshaler
  	}
  	}
  	}
  	// WithLogMarshalers adds additional log marshalers to the exporter factory.
  	func WithLogsMarshalers(logsMarshalers ...LogsMarshaler) FactoryOption {
  	return func(factory *kafkaExporterFactory) {
  	for _, marshaler := range logsMarshalers {
  	factory.logsMarshalers[marshaler.Encoding()] = marshaler
  	}
  	}
  	}

These little details make it so we can build a custom Marshaler, instantiate it, and wire it in the existing exporter without changing the exporter's code. I believe this would be the simplest approach while the Encoding Extension is being worked on

[atoulme]

Sure, this sounds like a valid enhancement. We eventually want to support the encoding extension. See #21152 and #28686

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• needs: Github issue template generation code needs this to generate the corresponding labels.
• exporter/awss3: @atoulme @pdelewski

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[aboguszewski-sumo]

Current issue status:
#23649 is ready to merge and covers the "marshaler" part. However, there are some changes needed regarding the path of the uploaded files: we'd like to be able to modify its format. That will probably be addressed in another issue.

[atoulme]

The PR was merged, the work is done - new issues to follow on encoding. We can close this issue.