inputs.docker plugin unix:///var/run/docker.sock Permission Denied

[dontobi]

Relevent telegraf.conf

[[inputs.docker]]
  endpoint = "/var/run/docker.sock"
  gather_services = false
  container_names = []
  source_tag = false
  container_name_include = []
  container_name_exclude = []
  timeout = "30s"
  perdevice = false
  perdevice_include = ["cpu"]
  total = true
  total_include = ["cpu", "blkio", "network"]
  tag_env = ["JAVA_HOME", "HEAP_SIZE"]
  docker_label_include = []
  docker_label_exclude = []

System info
Telegraf 1.20.3, RaspberryPiOS (Debian Buster) on Raspberry Pi 4, Docker-CE 20.10.10

Docker
docker-compose.yml

version: “3.8”
services:

telegraf:
    image: telegraf:1.20.3
    container_name: telegraf
    restart: unless-stopped
    environment:
      - HOST_ETC=/hostfs/etc
      - HOST_PROC=/hostfs/proc
      - HOST_SYS=/hostfs/sys
      - HOST_VAR=/hostfs/var
      - HOST_RUN=/hostfs/run
      - HOST_MOUNT_PREFIX=/hostfs
    security_opt:
      - apparmor=unconfined
    volumes:
      - /telegraf/telegraf.conf:/etc/telegraf/telegraf.conf:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /:/hostfs:ro
    networks:
      - InfluxDB
    depends_on:
      InfluxDB:
        condition: "service_healthy"

Steps to reproduce
Enable plugin in telegraf.conf
Start telegraf container
Observe errors in the log
No Docker timeseries data stored in InfluxDB

Expected behavior
Docker Engine API data collected through unix:///var/run/docker.sock and stored as timeseries to InfluxDB
Worked correctly in previous telegraf 1.20.x versions
Actual behavior

telegraf log error:
E! [inputs.docker] Error in plugin: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.21/containers/json?filters=%7B%22status%22%3A%5B%22running%22%5D%7D&limit=0": dial unix /var/run/docker.sock: connect: permission denied

Additional info
I assume that the bug is triggered by commit #536.

4ca4baf

[Bads125]

I have the same problem since watchtower upgrade my version to 1.20.3

[dontobi]

I have found a workaround. If you also use Portainer, change the container from user "Telegraf" to "root".

[powersj]

Hi,

We recently made a change to our Telegraf container images to run the telegraf process as the telegraf user and group. This means that when running Telegraf in a container, you will need to make sure that the telegraf user is added to the docker group.

The docker user directive can accomplish this by adding:

--user telegraf:$(stat -c '%g' /var/run/docker.sock)

We have a full post on why we made this change and the impact to users.

Thanks!

[pbranly]

Hi
Sorry if I am a noob.
I use telegraf as many others from a docker-compose.
What do I have to change just for telegraf ?
Thanks per advance
Philippebranly

[powersj]

Hi,

The Docker Compose setting uses the user option in the configuration YAML. For example if I want to share the Docker socket with the telegraf service:

services:
  telegraf:
      user: telegraf:<host_docker_gid>

Where GID is the GID of the socket itself on the host machine.

[pbranly]

Thank you
I have added user: telegraf:0
But after restart I have always:

E! [inputs.docker] Error in plugin: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.21/containers/json?filters=%7B%22status%22%3A%5B%22running%22%5D%7D&limit=0": dial unix /var/run/docker.sock: connect: permission denied
Phil

[powersj]

Can you run the following two commands on your host system (not in docker):

ls -l /var/run/docker.sock
stat -c '%g' /var/run/docker.sock

[pbranly]

Thank you for your help
Command 1

srw-rw---- 1 root docker 0 nov. 4 13:36 /var/run/docker.sock

Command 2:
998

Thanks
Phil

[powersj]

Based on that output the socket is owned by the docker group, which has group ID of 998. You will want to use that, not 0 in your compose file.

[pbranly]

Hey it works !
Thank very much
Regards from France
Phil

[martadinata666]

So any suggestion to use in swarm mode? as each node got different docker groups?

[philomory]

This doesn't really work in Kubernetes, either.

[rsuplina]

Hi @powersj Im hitting same issue but it still persists

ls -ln /var/run/docker.sock | awk '{print $4}'

1

  telegraf:
    image: telegraf:1.19
    container_name: telegraf
    user: telegraf:1
    volumes:
      - ./telegraf/telegraf.conf:/etc/telegraf/telegraf.conf:rw
      - /var/run/docker.sock:/var/run/docker.sock
    env_file:
      - .env
    depends_on:
      - influxdb

and my error

E! [inputs.docker] Error in plugin: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.21/info": dial unix /var/run/docker.sock: connect: permission denied

[y-a-n-n]

I'd like to bump this too... I'm seeing the exact same as rsuplina, above. I'm using docker desktop on mac, and see the following:

$ stat -x /var/run/docker.sock
  File: "/var/run/docker.sock"
  Size: 43           FileType: Symbolic Link
  Mode: (0755/lrwxr-xr-x)         Uid: (    0/    root)  Gid: (    1/  daemon)
Device: 1,18   Inode: 12309550    Links: 1
Access: Wed Oct 18 17:00:09 2023
Modify: Wed Oct 18 17:00:09 2023
Change: Wed Oct 18 17:00:09 2023
 Birth: Wed Oct 18 17:00:09 2023

I have added user: telegraf:1 to my docker-compose file but am still seeing the following in the logs of the telegraf container:

2023-10-24T05:22:30Z E! [inputs.docker] Error in plugin: permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D": dial unix /var/run/docker.sock: connect: permission denied

[Jacksaur]

Another bump from me. Both user: telegraf:990 (Docker Group) and telegraf:0 (Root) still result in permission errors.
This is also preventing the disk input from reading my disk usage.