Document how to run Agent and Fleet Server in docker

[dedemorton]

Summary

Document running Agent in Docker.

Document running Fleet Server in Docker.

• How is it different?
  • Non upgradable, with different paths [Elastic Agent] Fix Docker container to allow state to properly be handled beats#24817
• What environment variables exist?
• Provide an example setup for docker-compose

Thoughts

It makes sense to both DeDe and I to have all Elastic Agent Docker docs in one place. We can document the steps for running Elastic Agent on Docker and add a section about Fleet Server to that page.

Related

• Create a README file for the Docker container image Create a README for the docker container image #589.
• Add new topic about Fleet Server Add new topic about Fleet server #449 (this issue was copied from that meta issue)

[bmorelli25]

Brandon's notes

Spin it up

I was able to spin up Fleet Server in Docker with the help of the Fleet Server readme and an internal Slack thread. Here are the steps I followed:

1. Bootstrap Kibana

   cd kibana
   yarn kbn bootstrap
   

2. Add server.host: 0.0.0.0 to config/kibana.dev.yml to ensure Kibana is accessible to the container.

3. Open two terminals. Simultaneously:

   Start Elasticsearch from source, ensuring it's accessible to the container:

   nvm use
   yarn es snapshot -E xpack.security.authc.api_key.enabled=true -E http.host=0.0.0.0
   

   Start Kibana from source with no base path:

   nvm use
   yarn start --no-base-path
   

4. Wait.

5. Head to http://0.0.0.0:5601. Log in: elastic:changeme. Trigger Fleet setup by clicking Fleet or going to http://0.0.0.0:5601/app/fleet#/.

6. Determine your IP address by opening a new terminal window and running: ipconfig getifaddr en0.

7. Start Elastic Agent by running the following command. Replace {YOUR_IP} with the IP address from the previous step.

   docker run -e KIBANA_HOST=http://{YOUR_IP}:5601 -e KIBANA_USERNAME=elastic -e KIBANA_PASSWORD=changeme -e ELASTICSEARCH_HOST=http://{YOUR_IP}:9200 -e ELASTICSEARCH_USERNAME=elastic -e ELASTICSEARCH_PASSWORD=changeme -e KIBANA_FLEET_SETUP=1 -e FLEET_SERVER_ENABLE=1 -e FLEET_SERVER_INSECURE_HTTP=1 docker.elastic.co/beats/elastic-agent:7.13.0-SNAPSHOT
   

8. Head to http://0.0.0.0:5601/app/fleet#/fleet/agents and be happy because I see a single Healthy Agent.

Environment variables

Environment variables appear to be listed here: https://github.com/elastic/beats/blob/3ef90056bfb60b6d5e241c52d7ff9f9dd01baf30/x-pack/elastic-agent/pkg/agent/cmd/container.go#L61-L134. Need to chat with someone to learn more.

Docs

Copied from beats: https://gist.github.com/bmorelli25/e9600c4c7ba9880a15ec93f585cba55a

[dedemorton]

@peasead shared this link. You might find the details in the script here to be useful: https://github.com/peasead/elastic-container

[peasead]

I'm not sure if I overlooked it, but HTTPS is required when enrolling an agent in Fleet. If you don't use HTTPS, you get an error that says something like "error generated but error not recorded". Troubleshooting it was a 400 because the Agent was expecting https://IP:8220. I guessed through the issue based on this entry in the troubleshooting section.

The troubleshooting does say

By default, HTTPS protocol and port 8220 is expected by Fleet Server to communicate with Elasticsearch unless you have explicitly set it otherwise.

My suggestion would be to clearly define, when setting the Fleet Server host setting in Kibana that it must be HTTPS and that HTTPS (self-signed cert) is used by the Elastic Agent by default.

[dedemorton]

@peasead Did you try using the insecure flag? It looks like there is an env var called FLEET_INSECURE that lets connect to an HTTP server: https://github.com/elastic/beats/blob/3ef90056bfb60b6d5e241c52d7ff9f9dd01baf30/x-pack/elastic-agent/pkg/agent/cmd/container.go#L76. This is not a best practice though.

(The docs for env vars are being reviewed and will be merged soon)

[peasead]

@peasead Did you try using the insecure flag? It looks like there is an env var called FLEET_INSECURE that lets connect to an HTTP server: https://github.com/elastic/beats/blob/3ef90056bfb60b6d5e241c52d7ff9f9dd01baf30/x-pack/elastic-agent/pkg/agent/cmd/container.go#L76. This is not a best practice though.

(The docs for env vars are being reviewed and will be merged soon)

I did try the --insecure flag thinking the same thing, it would allow it to run without HTTPS. In fact, without the --insecure flag, I got an error saying I needed it.

Seemingly, the --insecure flag allows for a self-signed certificate? Similar to curl -k? I'm guessing here because without HTTPS it didn't work (even with the --insecure flag), but with HTTPS, it did work.

[bmorelli25]

Closed in #757.