Use Docker Secrets for Configuration Values

[JaronrH]

Discussed in #10706

^{Originally posted by JaronrH July 23, 2024}
I currently use Docker Swarm with Docker Secrets to store configuration information. I would like to load information from Secrets into my configuration. For example:

receivers:
  rabbitmq:
    endpoint: http://rabbitmq_rabbitmq:15672
    username: ${file:/run/secrets/RABBITMQ_USER}
    password: ${file:/run/secrets/RABBITMQ_PASSWORD}
    collection_interval: 10s

Docker Secrets are exposed as files in /run/secrets. The problem (as far as I can tell - I'm not a Go dev) is that the ${file:} processor is trying to load those secrets as yaml files instead of values! So, I'm getting errors like this as a result:

Error: failed to get config: cannot resolve the configuration: yaml: did not find expected node content
2024/07/23 16:08:53 collector server run finished with error: failed to get config: cannot resolve the configuration: yaml: did not find expected node content

Is there a better way to do this? Maybe some way to tell opentelemetry-collector-contrib to load everything in /run/secrets into env variables so I can use ${env:*}?

Thanks!

[TylerHelmuth]

@JaronrH what version of the collector are you using?

[TylerHelmuth]

/cc @mx-psi

[JaronrH]

@JaronrH what version of the collector are you using?

I'm using otelcol-contrib 0.105.0

[mx-psi]

@JaronrH could you try with 0.106.0?

[JaronrH]

@JaronrH could you try with 0.106.0?

I upgraded to 0.106.0 and tried it again. Here are my findings:

• Adding the RabbitMq config allowed the process to startup in Docker.
• Adding config for Postgres (see below) caused the same error to appear.
• RabbitMq receiver was getting 401/Unauthorized errors back from RabbitMq so I do not believe the credentials were actually being loaded from the Docker Secrets mount point.

  postgresql:
    endpoint: database_postgres:5432
    transport: tcp
    tls:
      insecure: true
    username: ${file:/run/secrets/dbUser}
    password: ${file:/run/secrets/dbPassword}

[mx-psi]

@JaronrH Thanks!!

Adding the RabbitMq config allowed the process to startup in Docker.

So to be clear about this, the following config works for you in v0.106.0 but does not work in v0.105.0?

receivers:
  rabbitmq:
    endpoint: http://rabbitmq_rabbitmq:15672
    username: ${file:/run/secrets/RABBITMQ_USER}
    password: ${file:/run/secrets/RABBITMQ_PASSWORD}
    collection_interval: 10s

Adding config for Postgres (see below) caused the same error to appear.

With this you mean the yaml: did not find expected node content error?

RabbitMq receiver was getting 401/Unauthorized errors back from RabbitMq so I do not believe the credentials were actually being loaded from the Docker Secrets mount point.

🤔 I think this part is likely not an issue of the Collector itself but some misconfiguration in the /run/secrets files.

[JaronrH]

So to be clear about this, the following config works for you in v0.106.0 but does not work in v0.105.0?

It "works" in that I did not get an error starting up. It does not seem to load the value from Docker Secrets as I'm getting the 401 error.

Adding config for Postgres (see below) caused the same error to appear.

With this you mean the yaml: did not find expected node content error?

That is correct!

RabbitMq receiver was getting 401/Unauthorized errors back from RabbitMq so I do not believe the credentials were actually being loaded from the Docker Secrets mount point.

🤔 I think this part is likely not an issue of the Collector itself but some misconfiguration in the /run/secrets files.

It very well could be! In the example from your documentation, which is loading a file for Exports, you're loading a yaml file that is used for the exports. In this case, the Docker Secrets file contains JUST the username (Ex: User) or JUST the password (Ex: Password123) and not a [partial] yaml file.

This is why I was wondering if there was a better way for me to be handling this in Docker OR if Docker Secrets support could be added?

For Example:

receivers:
  rabbitmq:
    endpoint: http://rabbitmq_rabbitmq:15672
    username: ${dockersecret:RABBITMQ_USER}
    password: ${dockersecret:RABBITMQ_PASSWORD}
    collection_interval: 10s

dockersecret: handler looks for a file in /run/secrets/ named RABBITMQ_USER and loads the files contents as the value to be inserted.

[mx-psi]

I am able to reproduce with a password like ["a", (I get the same error here: https://go.dev/play/p/aDSRTZ0tjl9).

The approach we are following since #10618 is to delay unmarshaling until we know the target field type. The problem here is that, even if we don't use the YAML output, we require the input to be parseable as YAML. We can fix that by also delaying the parsing, I can file a PR for that.

[mx-psi]

I am going to mark this as a release blocker. This has happened for a long time with the file provider, but this failure mode is relatively recent if you are using the ${..} syntax, so I consider this a regression.