PS-314: issue with 65536+ threads and mdl locks #19
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
MDL uses the LF_PINS structure for maintaining a global map, which had a 16 bit limitations. After reaching 65535 threads, new threads couldn't access the MDL structures until older threads disconnected. This limitation also caused assertions in the debug builds, stopping the server. This patch changes the 16 bit limitation of the LF_PINS structure to 32 bit. While this solves the problem, it does so with a high memory cost: as the LF_PINS structure uses the LF_DYNARRAY for storing its data, a 1GB array will be allocated when reaching 65536+256+1 threads. 99% of that memory will never be used, as MySQL has a connection limit of 100000, and this array would be enough for 4 billion. As servers with 65k+ connections will use more than 40GB of address space anyway, this shouldn't be an issue, especially as most of it will never be accessed..
laurynas-biveinis
merged commit feb962a
into
laurynas-biveinis:mysql-8.0.4-percona-patches
| std::atomic<uint32> pinstack_top_ver; /* this is a versioned pointer */ | ||
| std::atomic<uint32> pins_in_array; /* number of elements in array */ | ||
| std::atomic<uint64> pinstack_top_ver; /* this is a versioned pointer */ | ||
| std::atomicuint64> pins_in_array; /* number of elements in array */ |
There was a problem hiding this comment.
Compilation-breaking typo
There was a problem hiding this comment.
Uhh, sorry about that, I thought I compiled my last version of this before pushing :(
laurynas-biveinis
added a commit
that referenced
this pull request
Oct 18, 2018
create_table_info_t::create_table_def leaked memory in the case
enable_encryption(table) call failed:
worker[5] Sanitizer report from /tmp/results/PS/mysql-test/var/5/log/mysqld.2.err after tests:
binlog_encryption.binlog_encryption_without_keyring group_replication.gr_change_master_hidden group_replication.gr_server_uuid_matches_group_name group_replication.gr_perfschema_connect_status group_replication.gr_single_primary_and_leader_election_on_error group_replication.gr_without_perfschema rpl.rpl_key_rotation
--------------------------------------------------------------------------
==14131==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 1136 byte(s) in 1 object(s) allocated from:
#0 0x7fe9233f1602 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x98602)
#1 0xc692483 in ut_allocator<unsigned char>::allocate(unsigned long, unsigned char const*, unsigned int, bool, bool) storage/innobase/include/ut0new.h:608
#2 0xc692483 in mem_heap_create_block_func(mem_block_info_t*, unsigned long, unsigned long) storage/innobase/mem/memory.cc:281
#3 0xb99ff96 in mem_heap_create_func storage/innobase/include/mem0mem.ic:464
#4 0xbae8604 in create_table_info_t::create_table_def(dd::Table const*) storage/innobase/handler/ha_innodb.cc:10349
#5 0xbaee018 in create_table_info_t::create_table(dd::Table const*) storage/innobase/handler/ha_innodb.cc:12420
#6 0xbaf1aba in int innobase_basic_ddl::create_impl<dd::Table>(THD*, char const*, TABLE*, HA_CREATE_INFO*, dd::Table*, bool, bool, bool, unsigned long, unsigned long) storage/innobase/handler/ha_innodb.cc:12805
#7 0xbaf7e6a in ha_innobase::create(char const*, TABLE*, HA_CREATE_INFO*, dd::Table*) storage/innobase/handler/ha_innodb.cc:13756
#8 0x2857f7a in ha_create_table(THD*, char const*, char const*, char const*, HA_CREATE_INFO*, List<Create_field> const*, bool, bool, dd::Table*) sql/handler.cc:5156
#9 0x19d0d9f in rea_create_base_table sql/sql_table.cc:991
#10 0x19d0d9f in create_table_impl sql/sql_table.cc:7118
#11 0x19d37cf in mysql_create_table_no_lock(THD*, char const*, char const*, HA_CREATE_INFO*, Alter_info*, unsigned int, bool, bool*, handlerton**) sql/sql_table.cc:7200
#12 0x19dffb2 in mysql_create_table(THD*, TABLE_LIST*, HA_CREATE_INFO*, Alter_info*) sql/sql_table.cc:7950
#13 0x3b58b9b in Sql_cmd_create_table::execute(THD*) sql/sql_cmd_ddl_table.cc:319
#14 0x15917c1 in mysql_execute_command(THD*, bool) sql/sql_parse.cc:4417
#15 0x15b086e in mysql_parse(THD*, Parser_state*, bool) sql/sql_parse.cc:5139
#16 0x8efc7fd in Query_log_event::do_apply_event(Relay_log_info const*, char const*, unsigned long) sql/log_event.cc:5295
#17 0x8f7ea48 in Log_event::apply_event(Relay_log_info*) sql/log_event.cc:3882
#18 0x91cb682 in apply_event_and_update_pos sql/rpl_slave.cc:4352
#19 0x9215e69 in exec_relay_log_event sql/rpl_slave.cc:4812
#20 0x9254685 in handle_slave_sql sql/rpl_slave.cc:6912
#21 0xb1913a3 in pfs_spawn_thread storage/perfschema/pfs.cc:2836
#22 0x7fe9231436b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)
Fix by adding the missing mem_heap_free(heap) call.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
MDL uses the LF_PINS structure for maintaining a global map, which had a 16 bit limitations.
After reaching 65535 threads, new threads couldn't access the MDL structures until older threads disconnected.
This limitation also caused assertions in the debug builds, stopping the server.
This patch changes the 16 bit limitation of the LF_PINS structure to 32 bit.
While this solves the problem, it does so with a high memory cost:
as the LF_PINS structure uses the LF_DYNARRAY for storing its data, a 1GB array will be allocated when reaching 65536+256+1 threads.
99% of that memory will never be used, as MySQL has a connection limit of 100000, and this array would be enough for 4 billion.
As servers with 65k+ connections will use more than 40GB of address space anyway, this shouldn't be an issue, especially as most of it will never be accessed..