Docker Swarm: No Data on Telegraf Syslog Input

[turbo]

Relevant telegraf.conf:

# Read metrics about CPU usage
[[inputs.cpu]]
  percpu = false
  totalcpu = true
  fieldpass = [ "usage*" ]
  name_suffix = "_vm"

# Read metrics about disk usagee
[[inputs.disk]]
  fielddrop = [ "inodes*" ]
  mount_points=["/"]
  name_suffix = "_vm"

# Read metrics about network usage
[[inputs.net]]
  interfaces = [ "eth0" ]
  fielddrop = [ "icmp*", "ip*", "tcp*", "udp*" ]
  name_suffix = "_vm"

# Read metrics about memory usage
[[inputs.mem]]
  name_suffix = "_vm"

# Read metrics about swap memory usage
[[inputs.swap]]
  name_suffix = "_vm"

# Read metrics about system load & uptime
[[inputs.system]]
  name_suffix = "_vm"

# Read metrics from docker socket api
[[inputs.docker]]
  endpoint = "unix:///var/run/docker.sock"
  container_names = []
  name_suffix = "_docker"

[[inputs.syslog]]
  server = "udp://localhost:6514"

[[outputs.influxdb]]
  database = "vm_metrics"
  urls = ["http://influxdb:8086"]
  namepass = ["*_vm"]

[[outputs.influxdb]]
  database = "docker_metrics"
  urls = ["http://influxdb:8086"]
  namepass = ["*_docker"]

System info:

Here is the complete stack:

version: "3.6"

##################
#   Extensions   #
##################

x-influxlog: &influxlog
  logging:
    driver: syslog
    options:
      syslog-address: "udp://159.89.101.158:6514"
      syslog-format: rfc5424micro
      tag: "staging-{{.ImageName}}"

x-worker: &worker
  placement:
    constraints: [node.role == worker]

x-manager: &manager
  placement:
    constraints: [node.role == manager]

#################
#    Configs    #
#################

configs:
  telegraf-config:
    external: true
  kapacitor-config:
    external: true

##################
#    Networks    #
##################

networks:
  traefik-net:
    external: true
  tick-net:
    driver: overlay

#################
#    Volumes    #
#################

volumes:
  influxdb:
    external: true
  chronograf:
    external: true
  kapacitor:
    external: true

##################
#    Services    #
##################

services:
  # Service:        Telegraf
  # Purpose:        Data collection agent
  # Documentation:  https://docs.influxdata.com/telegraf
  telegraf:
    image: telegraf
    networks:
      - tick-net
    ports:
      - 6514:6514
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    configs:
      - source: telegraf-config
        target: /etc/telegraf/telegraf.conf
    deploy:
      mode: global
      restart_policy:
        condition: on-failure

  # Service:        InfluxDB
  # Purpose:        Time-series database
  # Documentation:  https://docs.influxdata.com/influxdb
  influxdb:
    image: influxdb
    networks:
      - tick-net
    volumes:
      - influxdb:/var/lib/influxdb
    deploy:
      <<: *manager
      restart_policy:
        condition: on-failure

  # Service:        Chronograf
  # Purpose:        InfluxDB frontend
  # Documentation:  https://docs.influxdata.com/chronograf
  chronograf:
    image: chronograf
    environment:
      INFLUXDB_URL: http://influxdb:8086
      KAPACITOR_URL: http://kapacitor:9092
    networks:
      - tick-net
      - traefik-net
    volumes:
      - chronograf:/var/lib/chronograf
    <<: *influxlog
    deploy:
      <<: *manager
      restart_policy:
        condition: on-failure
      labels:
        - traefik.enable=true
        - traefik.docker.network=traefik-net
        - traefik.frontend.rule=Host:tick.[REDACTED]
        - traefik.port=8888
        - traefik.frontend.auth.basic=admin:[REDACTED]

  # Service:        Kapacitor
  # Purpose:        ETL and alerting for InfluxDB
  # Documentation:  https://docs.influxdata.com/kapacitor
  kapacitor:
    image: kapacitor
    environment:
      KAPACITOR_HOSTNAME: kapacitor
      KAPACITOR_INFLUXDB_0_URLS_0: http://influxdb:8086
    networks:
      - tick-net
    volumes:
      - kapacitor:/var/lib/kapacitor
    configs:
      - source: kapacitor-config
        target: /etc/kapacitor/kapacitor.conf
    deploy:
      <<: *manager
      restart_policy:
        condition: on-failure

Steps to reproduce:

1. Deploy the stack

Expected behavior:

• syslog table autogenerated, log messages from chronograf service available in log viewer

Actual behavior:

• No logging input (syslog tables absent also in Data Explorer)

Additional info:

Docker doesn't complain about the server being impossible to contact, because the protocol is set to UDP. Previous discussion at #4482. Solutions suggested by @glinton and @danielnelson both don't work.

[turbo]

Additonal Info 1:

The dummy TLS config by @glinton keeps the chronograf service from starting alltogether:

failed to initialize logging driver: EOF

[turbo]

Progress:

I have setup rsyslog on the manager:

$WorkDirectory /tmp/rsyslog
$ActionQueueType LinkedList
$ActionQueueFileName srvrfwd
$ActionResumeRetryCount -1
$ActionQueueSaveOnShutdown on
$ModLoad imudp
$UDPServerAddress localhost $UDPServerRun 514 *.* 
@@(o)127.0.0.1:6514;RSYSLOG_SyslogProtocol23Format

I have changed to telegraf config to bind on all interfaces and added an additional output:

[[inputs.syslog]]
  server = "tcp://0.0.0.0:6514"
  name_suffix = "_sysl"

# ...

[[outputs.influxdb]]
  database = "syslog"
  urls = ["http://influxdb:8086"]
  namepass = ["*_sysl"]

I have changed my logging extension to:

x-influxlog: &influxlog
  logging:
    driver: syslog
    options:
      syslog-address: "udp://localhost:514"
      syslog-format: rfc5424micro
      tag: "{{.ImageName}}"

Now, in the data explorer, I can see the syslog table and query it manually:

SELECT "message", "severity_code" FROM "syslog"."autogen"."syslog_sysl" WHERE time > now() - 1h

Remaining problems:

• Logs don't appear in the log viewer, because of the custom suffix (the hardcoded query of LV points to +.syslog, not +.syslog_sysl, naturally). But if I take away the name and namepass config in telegraf, the contents of the syslog table is replaced by the output of all inputs.
• The set tag does not appear within the logs, appname is now just "dockerd". I suspect this has to do with the rsyslog proxy.

[turbo]

Progress 2:

I've changed the Telegraf config as follows:

[[inputs.syslog]]
  server = "tcp://0.0.0.0:6514"

# ...

[[outputs.influxdb]]
  database = "sys"
  urls = ["http://influxdb:8086"]
  namepass = ["syslog"]

Now logs appear in log viewer. However, only dockerd and other system logs, no entries related to the docker container chronograf, which is supposed to log.

[turbo]

Update yet again. The Rsyslog config was bogus (taken from the Influx blog). The new syntax configures the UDP module directly. Only the last line for the proxy is needed.

Configure Rsyslog

Uncomment and modify:

module(load="imudp")
input(type="imudp" port="514" address="*")

Append:

@@(o)127.0.0.1:6514;RSYSLOG_SyslogProtocol23Format

Configure Telegraf

[[inputs.syslog]]
  server = "tcp://0.0.0.0:6514"

# ...

[[outputs.influxdb]]
  database = "sys"
  urls = ["http://influxdb:8086"]
  namepass = ["syslog"]

Configure Your Stack

x-influxlog: &influxlog
  logging:
    driver: syslog
    options:
      syslog-address: "udp://localhost:514"
      tag: "{{.ImageName}}"

(i.e. don't set the msg format!)

Enjoy

[maxlein]

As you appended two @@ and are using server = "tcp://0.0.0.0:6514", you are using tcp for logging and not udp!?