Metricbeat rabbitmq module issues

[adriananeci]

Hi,

Based on https://discuss.elastic.co/t/metricbeat-rabbitmq-module-issues/125751 there could be a possible bug in rabbitmq metrcibeat module.
I' running metricbeat version 6.2.3 inside a docker container. Also rabbitmq is running inside a container, using an official image, 3.7-management (current version is 3.7.4 and Erlang 20.2.4). Docker is running on top of CentOS 7.4.

Steps to reproduce:

• start a redis instance to be used as output for metricbeat:
  docker run --name metricbeat-redis -p 6381:6379 -d redis redis-server --requirepass 1234
• start a rabbitmq instance inside a container using rabbitmq:3.7-management image like:
  docker run -h 'rabbit-1' -m 0b -p 13000:5672 -p 13001:15672 -p 13002:13002 -p 13003:13003 -e RABBITMQ_DIST_PORT=13002 -e ERL_EPMD_PORT=13003 -e RABBITMQ_ERLANG_COOKIE=some-pasword -e RABBITMQ_DEFAULT_USER=user -e RABBITMQ_DEFAULT_PASS=password --name rabbit-1 -d rabbitmq:3.7-management
• start metricbeat 6.2.3 inside a container and configure rabbitmq module in modules.d to connect to the rabbitmq local node:
  docker run --net host -m 1g -v /:/hostfs:ro -v /proc:/hostfs/proc:ro -v /sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro -v /tmp/metricbeat.yml:/usr/share/metricbeat/metricbeat.yml -v /tmp/modules.d:/usr/share/metricbeat/modules.d --pid=host --name metricbeat docker.elastic.co/beats/metricbeat:6.2.3 metricbeat -e -system.hostfs=/hostfs -E name=lab.local

where metricbeat.yml looks like:

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.period: 10s
  reload.enabled: true

name: lab.local

output.redis:
  enabled: true
  hosts: ["localhost:6381"]
  
  key: logstash

  password: 1234

and rabbitmq.yml:

- module: rabbitmq
  # connection metric will be available in a future metricbeat release. not working with latest tested version, 6.2.3
  metricsets: ["node", "queue"] #, "connection"]
  period: 10s
  hosts: ["localhost:13001"]
  
  username: 'user'
  password: 'password'

• create a queue in the default rabbitmq vhost using rabbitmq management UI or other methods.
• Send a message to that queque (optional step)
• check metricbeat logs:

2018-03-28T13:29:46.488Z	ERROR	schema/schema.go:41	Error on field 'count': Missing field: count, Error: Key disk_reads not found
2018-03-28T13:29:46.488Z	ERROR	schema/schema.go:41	Error on field 'count': Missing field: count, Error: Key disk_writes not found
2018-03-28T13:29:57.142Z	ERROR	schema/schema.go:41	Error on field 'count': Missing field: count, Error: Key disk_reads not found
2018-03-28T13:29:57.143Z	ERROR	schema/schema.go:41	Error on field 'count': Missing field: count, Error: Key disk_writes not found
2018-03-28T13:30:03.247Z	INFO	[monitoring]	log/log.go:124	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":880,"time":889},"total":{"ticks":2700,"time":2709,"value":2700},"user":{"ticks":1820,"time":1820}},"info":{"ephemeral_id":"0b880f5f-40cf-4ec0-a55b-b549c4f887b3","uptime":{"ms":4740265}},"memstats":{"gc_next":4526032,"memory_alloc":2645784,"memory_total":226571184}},"libbeat":{"config":{"module":{"running":1},"reloads":3},"output":{"events":{"acked":6,"batches":3,"total":6},"read":{"bytes":21},"write":{"bytes":5328}},"pipeline":{"clients":1,"events":{"active":0,"published":6,"total":6},"queue":{"acked":6}}},"metricbeat":{"rabbitmq":{"node":{"events":3,"success":3},"queue":{"events":3,"success":3}}},"system":{"load":{"1":0,"15":0.05,"5":0.02,"norm":{"1":0,"15":0.0125,"5":0.005}}}}}}
2018-03-28T13:30:06.594Z	ERROR	schema/schema.go:41	Error on field 'count': Missing field: count, Error: Key disk_writes not found
2018-03-28T13:30:06.594Z	ERROR	schema/schema.go:41	Error on field 'count': Missing field: count, Error: Key disk_reads not found

Could you please have a look and confirm if indeed is a bug or a wrong configuration on my side?
I've also activated debug logs for metricbeat (-d '*') but didn't find anything relevant.

[jsoriano]

Disk entries were made optional in #6876, this should fix these issues.