docker logs env variables

[kpweiler]

Current Vector Version

vector 0.11.1 (v0.11.1 x86_64-unknown-linux-musl 2020-12-17)

Use-cases

I'm trying to run vector as a general purpose log shipper for a Nomad cluster (with Docker execution). Ideally - I'd like log shipping and aggregation to be totally transparent to the user. In order to accomplish this - metadata about each docker container needs to be collected.

Attempted Solutions

The canonical solution to this problem is to enrich your images (or containers, at runtime) with docker labels and parse the labels in the log aggregator. The docker daemon on each Nomad client node needs to be configured to pass through label metadata to the log driver - like this (in daemon.json):

{
  "log-driver": "json-file",
  "log-opts": {
    "labels": "logging_job_name"
  }
}

This configuration instructs docker to forward any labels whose key is logging_job_name along with the other log metadata to the json-file log driver (by default).
Vector has support for this process in that the Vector docker source forwards label metadata to downstream transforms and sinks. This solution, however, requires each user to label the job with either information that already exists in the container as environment variables by default (NOMAD_JOB_NAME, for example), or create labels that might clash with other job's labels. Nomad has no way to enforce unique labels per job/task.

Proposal

In addition to the above docker configuration for labels, docker can also optionally forward environment variable key/values with the log metadata. The configuration looks quite similar:

{
  "log-driver": "json-file",
  "log-opts": {
    "env": "NOMAD_JOB_NAME"
  }
}

This daemon configuration could work in concert with vector configuration in the following way - I've used the file sync as a simple example:

[sources.dockerd]
  type = "docker_logs"

[sinks.file]
  type = "file"
  inputs = ["dockerd"]
  encoding.codec = "text"
  path = "/q/logs/prefect/{{env.NOMAD_JOB_NAME}}/%Y-%m-%d/stdout.log"

Of course - this environment variable could also be consumed by downstream vector transforms, sinks, or log aggregators.

References

I couldn't find any tickets like this, but I did have a brief discussion with @jszwedko on the discord chat.

The relevant bit of code in vector to add to probably starts with this struct:
https://github.com/timberio/vector/blob/master/src/sources/docker_logs.rs#L65-L78

[kpweiler]

For full disclosure - this PR would obviate the need for this for my use-case, but I still think it's a worthwhile enhancement in that it more fully aligns with the docker logs API.