Crash on mysql 5.7.17-1 when performace_schema=0;

[dan-po]

Hello.

I'm qualifying McAfee MySQL auditing for my organization using MySQL Community Edition v. 5.7.17-1
McAffee SQL auditing version: audit-plugin-mysql-5.7-1.1.3-705-linux-x86_64

My configuration parameters are as follows:

plugin-load=AUDIT=libaudit_plugin.so
audit_json_log_file = /mysql/logs/mysql_audit.log
audit_json_file = ON
audit_uninstall_plugin = ON
audit_whitelist_users = '{},...'

This works fine typically, however as part of performance benchmarking, I'm starting mysql with the performance_schema turned off (default is on in 5.7) by adding the following to my.cnf:

performance_schema=OFF

Whenever both performance_schema=OFF and autidt_json_file=ON, mysqld crashes with the following stack trace:

19:26:32 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.

key_buffer_size=12582912
read_buffer_size=2097152
max_used_connections=10
max_threads=6000
thread_count=6
connection_count=4
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 18524350 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7fc40c00aed0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fc49b31ae28 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x35)[0xf42a95]
/usr/sbin/mysqld(handle_fatal_signal+0x4a4)[0x7cd8b4]
/lib64/libpthread.so.0(+0xf7e0)[0x7fd1b06367e0]
/usr/lib64/mysql/plugin/libaudit_plugin.so(+0x132e1)[0x7fc4a140c2e1]
/usr/lib64/mysql/plugin/libaudit_plugin.so(+0x11768)[0x7fc4a140a768]
/usr/lib64/mysql/plugin/libaudit_plugin.so(+0x127d7)[0x7fc4a140b7d7]
/usr/lib64/mysql/plugin/libaudit_plugin.so(+0x12931)[0x7fc4a140b931]
/usr/lib64/mysql/plugin/libaudit_plugin.so(+0xec91)[0x7fc4a1407c91]
/usr/lib64/mysql/plugin/libaudit_plugin.so(+0x10668)[0x7fc4a1409668]
/usr/sbin/mysqld(_Z11mysql_parseP3THDP12Parser_state+0x3a5)[0xd0ed95]
/usr/sbin/mysqld(_Z16dispatch_commandP3THDPK8COM_DATA19enum_server_command+0x1778)[0xd10578]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x194)[0xd10ea4]
/usr/sbin/mysqld(handle_connection+0x29c)[0xde3f1c]
/usr/sbin/mysqld(pfs_spawn_thread+0x174)[0xf60b74]
/lib64/libpthread.so.0(+0x7aa1)[0x7fd1b062eaa1]
/lib64/libc.so.6(clone+0x6d)[0x7fd1af18cbcd]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7fc40c012830): is an invalid pointer
Connection ID (thread ID): 245
Status: NOT_KILLED

I see that there's a newer version as of 20 days ago, I'll start by upgrading.

[dan-po]

Crashes under audit-plugin-mysql-5.7-1.1.4-725-linux-x86_64 as well:

19:51:22 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.

key_buffer_size=12582912
read_buffer_size=2097152
max_used_connections=0
max_threads=6000
thread_count=3
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 18524350 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x58293fe0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fcca539ec28 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x35)[0xf42a95]
/usr/sbin/mysqld(handle_fatal_signal+0x4a4)[0x7cd8b4]
/lib64/libpthread.so.0(+0xf7e0)[0x7fd9b9c897e0]
/usr/lib64/mysql/plugin/libaudit_plugin.so(+0x13511)[0x7fccac262511]
/usr/lib64/mysql/plugin/libaudit_plugin.so(+0x118f8)[0x7fccac2608f8]
/usr/lib64/mysql/plugin/libaudit_plugin.so(+0x12887)[0x7fccac261887]
/usr/lib64/mysql/plugin/libaudit_plugin.so(+0x129e1)[0x7fccac2619e1]
/usr/lib64/mysql/plugin/libaudit_plugin.so(+0xed81)[0x7fccac25dd81]
/usr/lib64/mysql/plugin/libaudit_plugin.so(+0x107b8)[0x7fccac25f7b8]
/usr/sbin/mysqld(_Z11mysql_parseP3THDP12Parser_state+0x3a5)[0xd0ed95]
/usr/sbin/mysqld[0x7eacc1]
/usr/sbin/mysqld(handle_bootstrap+0xb4)[0x7eb304]
/usr/sbin/mysqld(pfs_spawn_thread+0x174)[0xf60b74]
/lib64/libpthread.so.0(+0x7aa1)[0x7fd9b9c81aa1]
/lib64/libc.so.6(clone+0x6d)[0x7fd9b87dfbcd]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (58289880): SELECT TABLE_SCHEMA, TABLE_NAME FROM INFORMATION_SCHEMA.TABLES WHERE CREATE_OPTIONS LIKE '%partitioned%';
Connection ID (thread ID): 3
Status: NOT_KILLED

[aharonrobbins]

Thanks for the information. We will try to reproduce this and hopefully come up with a fix.

[dan-po]

Any updates? Can I assist w/ anything?

[aharonrobbins]

Hi. Sorry - no updates. I've not had time to investigate. You should be able to use addr2line to determine in which routine it's crashing. Then, if you are handy with a debugger, please compile the plugin for debugging, attach gdb to the running mysqld, set a breakpoint in the crashing routine, and see why it crashes. Thanks!

[aharonrobbins]

Hi. Please see if the attached diff fixes the problem for you. It does for me. Thanks!
performance-schema-0-fix.diff.txt

[dan-po]

Any chance you can provide a binary build with these changes? It would save me a bunch of time.

dmp

[aharonrobbins]

We will eventually. But I can't commit to a schedule at the moment. This has to go through our internal QA process. I will try to expidite it.

[dan-po]

Awesome, thanks. I wont have any cycles this week to build a build environment, perhaps towards the end of August.

[aharonrobbins]

OK. I've got this into our internal process, so I can hope it'll be in a dev-snapshot relatively soon.

[aharonrobbins]

The fix for this issue has been pushed to Github and new binaries uploaded to bintray.com (dev-snapshot). Closing this issue. Enjoy.