README.md

AWS S3 Exporter for OpenTelemetry Collector

Status
Stability	alpha: traces, metrics, logs
Distributions	contrib
Issues
Code Owners	@atoulme, @pdelewski

Schema supported

This exporter targets to support proto/json format.

Exporter Configuration

The following exporter configuration parameters are supported.

Name	Description	Default
region	AWS region.	"us-east-1"
s3_bucket	S3 bucket
s3_prefix	prefix for the S3 key (root directory inside bucket).
s3_partition_format	filepath formatting for the partition; See strftime for format specification.	"year=%Y/month=%m/day=%d/hour=%H/minute=%M"
role_arn	the Role ARN to be assumed
file_prefix	file prefix defined by user
marshaler	marshaler used to produce output data	otlp_json
encoding	Encoding extension to use to marshal data. Overrides the marshaler configuration option if set.
encoding_file_extension	file format extension suffix when using the encoding configuration option. May be left empty for no suffix to be appended.
endpoint	(REST API endpoint) overrides the endpoint used by the exporter instead of constructing it from region and s3_bucket
storage_class	S3 storageclass	STANDARD
acl	S3 Object Canned ACL	private
s3_force_path_style	set this to true to force the request to use path-style addressing	false
disable_ssl	set this to true to disable SSL when sending requests	false
compression	should the file be compressed	none
sending_queue	exporters common queuing	disabled

Marshaler

Marshaler determines the format of data sent to AWS S3. Currently, the following marshalers are implemented:

• otlp_json (default): the OpenTelemetry Protocol format, represented as json.
• otlp_proto: the OpenTelemetry Protocol format, represented as Protocol Buffers. A single protobuf message is written into each object.
• sumo_ic: the Sumo Logic Installed Collector Archive format. This format is supported only for logs.
• body: export the log body as string. This format is supported only for logs.

Encoding

Encoding overrides marshaler if present and sets to use an encoding extension defined in the collector configuration.

See https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/encoding.

Compression

• none (default): No compression will be applied
• gzip: Files will be compressed with gzip. This does not support sumo_icmarshaler.

Example Configurations

Following example configuration defines to store output in 'eu-central' region and bucket named 'databucket'.

exporters:
  awss3:
    s3uploader:
      region: 'eu-central-1'
      s3_bucket: 'databucket'
      s3_prefix: 'metric'

    # Optional (disabled by default)
    sending_queue:
      enabled: true
      num_consumers: 10
      queue_size: 100

Logs and traces will be stored inside 'databucket' in the following path format.

metric/year=YYYY/month=MM/day=DD/hour=HH/minute=mm

Partition Formatting

By setting the s3_partition_format option, users can specify the file path for their logs. See the strftime reference for more formatting options.

exporters:
  awss3:
    s3uploader:
      region: 'eu-central-1'
      s3_bucket: 'databucket'
      s3_prefix: 'metric'
      s3_partition_format: '%Y/%m/%d/%H/%M'

In this case, logs and traces would be stored in the following path format.

metric/YYYY/MM/DD/HH/mm

AWS Credential Configuration

This exporter follows default credential resolution for the aws-sdk-go.

Follow the guidelines for the credential configuration.

OpenTelemetry Collector Helm Chart for Kubernetes

For example, when using OpenTelemetry Collector Helm Chart you could use extraEnvs in the values.yaml.

extraEnvs:
- name: AWS_ACCESS_KEY_ID
  value: "< YOUR AWS ACCESS KEY >"
- name: AWS_SECRET_ACCESS_KEY
  value: "< YOUR AWS SECRET ACCESS KEY >"