meta: Docker log driver plugin, explerimental to GA

[fearful-symmetry]

We have a branch on feature/dockerbeat. Name is TBD.

This is a docker logging plugin that's installed via docker plugin install... and works alongside the docker engine to send logs to elasticsearch. It uses libbeat and supports most of the config options and outputs libbeat does. Initial commit was #13761

Issues

Experimental Release - 7.6

• We need an actual name

• internal logging is a mess

• Send out container tags [Docker log driver] cleanup output #14615

• The code that reads from the FIFO pipe is a mess, and copied from docker's example plugin. The protobuf reader is questionable, and we might want to implement our own reader. [docker plugin] Refactor pipeline reader for data safety #14375

• remove trailing \r from log lines [Docker log driver] cleanup output #14615

• Potential race with creating new pipelines [docker log driver] Move pipeline check + creation to single atomic operation #14518

• Go version used on master is enforcing import format comments, breaking docker/engine [Docker logging driver] Shuffle around docker deps due to import path checking #14523

• We need a test suite [Docker Log driver] Expand testing, add mocking tooling #14828

• make check and make fmt support?

• Use LoadWithSettings [Elastic Logging Plugin] Use LoadWithSettings and clean up pipeline code #15038

• Use proper dedot/stripping functions

• We need to figure out how to get on the docker plugin store

• Clean up vendor/ and deal with conflicting docker/prometheus dependencies

• Proper asciidocs working on this with @dedemorton [docs] Add early draft of Elastic Log Driver docs #15799

7.9.0

• Address remaining docs TODOs in meta: Docker log driver plugin, explerimental to GA #13990
• Create a config shim [Elastic Log Driver] Create a config shim between libbeat and the user #18605
• docker logs needs to work. This may depend on the spooling features.

7.10.0

• ECS compliance and integration with Kibana Logs
• Docker certification

Later

• We need to make some kind of decision about how we'll drop events. If the plugin's FIFO queues get backed up, the container's writes to stdio will block.
• Create an ingest pipeline manager to ingest documents based on container tags and names. This reduces the need for addtional config flags. Designing this with @urso .
• Parsing of JSON messages
• Multiline support
• Can we get this to send its own logs / health data to ES?
• some kind of integration test suite.
• Update documentation in config.json

Testing

This is fairly easy to test.

Once you have everything pulled down, run mage buildAndInstall to build the plugin. Docker plugins are configured with --log-opts, you can start a plugin using the log handler with:

$ docker run --log-driver=elastic/elastic-log-driver:8.0.0 \
--log-opt output.elasticsearch.hosts="172.18.0.2:9200" \
--log-opt output.elasticsearch.index="dockerbeat-test" \ 
-it debian:jessie /bin/bash`

Almost any option that you would specify with -E in filebeat, you could specify with --log-opt in the plugin.

[andresrc]

Given that this introduces a new artifact, we should also include here the integration in the build pipeline and the release process (/cc @exekias )

[Gavin89]

This would be a great addition, and would help push logs straight to ES from docker, without having to use Logstash.

[exekias]

I just saw the docker logs command part. That sounds a bit complex, perhaps we could move it to be done after the initial experimental release? cc @sorantis

[elasticmachine]

Pinging @elastic/integrations-services (Team:Services)

[jlind23]

@fearful-symmetry any chance you remember what we should do with this one?

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[fearful-symmetry]

@jlind23 it's technically GA, but heavily on the backburner, due to...everything going on, and also the extreme limitations of docker's plugin API.

[jlind23]

@fearful-symmetry thanks, then i'll close this one :)