[BUG] Error monitoring MySQL - list index out of range

[rbernardes]

Agent Environment
Agent 7.42.0 - Commit: 1b72bcf - Serialization version: v5.0.46 - Go version: go1.18.9

Describe what happened:
ERROR | (pkg/collector/worker/check_logger.go:69 in Error) | check:mysql | Error running check: [{"message": "list index out of range", "traceback": "Traceback (most recent call last):\n File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/base.py\", line 1122, in run\n self.check(instance)\n File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/mysql/mysql.py\", line 248, in check\n raise e\n File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/mysql/mysql.py\", line 229, in check\n self._collect_metrics(db, tags=tags)\n File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/mysql/mysql.py\", line 378, in _collect_metrics\n results.update(self.innodb_stats.get_stats_from_innodb_status(db))\n File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/mysql/innodb_metrics.py\", line 207, in get_stats_from_innodb_status\n results['Innodb_pending_log_flushes'] = long(row[4])\nIndexError: list index out of range\n"}]
Describe what you expected:
Monitoring InnoDB data from mariadb

Steps to reproduce the issue:
initi_config

instances:
server: 127.0.0.1
user: datadog
pass: "pass" # from the CREATE USER step earlier
port: "3306" # e.g. 3306
options:
replication: false
galera_cluster: true
extra_status_metrics: true
extra_innodb_metrics: true
extra_performance_metrics: true
schema_size_metrics: false
disable_innodb_metrics: false

Additional environment details (Operating System, Cloud provider, etc):
Ubuntu 20.04
Kernel 5.4.0-136-generic
MariaDB 10.10.2

[thedotedge]

Same issue happening with mariadb 10.11.2 and agent v7.43.0.

[meklon-imhio]

Similar issue:
datadog-agent 7.41.0
MariaDB 10.9.4

Oracle Linux 7
5.4.17-2136.315.5.el7uek.x86_64 #2 SMP Wed Dec 21 19:57:57 PST 2022 x86_64 x86_64 x86_64 GNU/Linux

Instance #mysql:fac024d1e8751dc3[ERROR]: [{"message": "list index out of range", "traceback": "Traceback (most recent call last):\n File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/base.py\", line 1122, in run\n self.check(instance)\n File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/mysql/mysql.py\", line 248, in check\n raise e\n File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/mysql/mysql.py\", line 229, in check\n self._collect_metrics(db, tags=tags)\n File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/mysql/mysql.py\", line 378, in _collect_metrics\n results.update(self.innodb_stats.get_stats_from_innodb_status(db))\n File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/mysql/innodb_metrics.py\", line 207, in get_stats_from_innodb_status\n results['Innodb_pending_log_flushes'] = long(row[4])\nIndexError: list index out of range\n"}]

[thedotedge]

Will anyone from maintainers be able to at least triage the issue?

[parasolx]

Similar issue:
datadog-agent 7.43.1
MariaDB 10.9.5
Linux CentOS 8 Stream

[jgb]

This is happening also on mariadb v10.11.2, ubuntu 22.04 LTS and datadog 7.43.1.
Can this be acknowledged because it completely breaks any mysql / mariadb monitoring on datadog...

[marnixgb]

This needs fixing, it was fine with mariadb 10.6.11 but having moved to 10.11.2 it also break in that way

Rocky 8.7
datadog-agent 7.42 or 7.43.1
MariaDB 10.11.2

This will become an increasing problem as 10.11.2 went GA LTS just recently

[marnixgb]

This problem exists because of a change in the output of : SHOW ENGINE INNODB STATUS;

Rocky 8.7
datadog-agent 7.43.1
MariaDB 10.11.2

The datadog agent under previous versions of MariaDB decodes the following output from that command :

Pending flushes (fsync) log: 0; buffer pool: 0
239932765 OS file reads, 105416645 OS file writes, 23950402 OS fsyncs
1 pending reads, 0 pending writes
55.59 reads/s, 16148 avg bytes/read, 19.50 writes/s, 9.10 fsyncs/s

The new output looks like this :

Pending flushes (fsync): 0
1999684 OS file reads, 14115 OS file writes, 14213 OS fsyncs
172.57 reads/s, 15790 avg bytes/read, 1.00 writes/s, 1.17 fsyncs/s

In /opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/mysql/innodb_metrics.py line 205 :

            elif line.find('Pending flushes (fsync)') == 0:
                # Pending flushes (fsync) log: 0; buffer pool: 0
                results['Innodb_pending_log_flushes'] = long(row[4])
                results['Innodb_pending_buffer_pool_flushes'] = long(row[7])

Is where the issue lies

A workaround for this is to change that section to the following (with a loss of some stats but the agent will at least work for key indicators like replication lag) :

           elif line.find('Pending flushes (fsync)') == 0:
                # Pending flushes (fsync) log: 0; buffer pool: 0
                #results['Innodb_pending_log_flushes'] = long(row[4])
                results['Innodb_pending_log_flushes'] = 0
                #results['Innodb_pending_buffer_pool_flushes'] = long(row[7])
                results['Innodb_pending_buffer_pool_flushes'] = 0

I'll report this issue through the support channel also so hopefully it will get some attention

[DiveSurfer]

I can also confirm this, for:

• Ubuntu 22.04.2 LTS
• Agent 7.43.1
• 10.11.2-MariaDB

An easier workaround is to set "disable_innodb_metrics" to true, then "only" the InnoDB metrics are missing.

[marnixgb]

There is an optional mysql integration package available which solves this issue, to install on centos/rhel/rocky do the following (cannot be run as root though - the below allow it run run as datadog agent while executing as root as dd-agent has no shell) :

su --shell=/bin/bash --session-command="export TERM=xterm;datadog-agent integration install datadog-mysql==10.2.0" dd-agent

systemctl restart datadog-agent

datadog-agent check mysql

[aweaver89]

Also, if needed, there is this page dedicated to the commands needed to install this version. The key is being able to run something similar to sudo datadog-agent integration install mysql==10.2.0

These changes were just implemented last week after work done on this PR

Hope this helps get everyone back up and running!