Thread Deadlock with proxysql CLuster #3694
Comments
|
how to reproduce: Docker-compose.yamlversion: '3'
services:
proxysql1:
container_name: proxysql1
build: .
restart: always
networks:
proxysql_net:
proxysql2:
container_name: proxysql2
build: .
restart: always
networks:
proxysql_net:
networks:
proxysql_net:
ipam:
driver: default
DockerfileFROM ubuntu:18.04
RUN apt update && apt install -y wget gdb mysql-client iproute2 inetutils-ping
RUN wget https://github.com/sysown/proxysql/releases/download/v2.3.2/proxysql_2.3.2-dbg-ubuntu18_amd64.deb && \
dpkg -i proxysql_2.3.2-dbg-ubuntu18_amd64.deb
ADD proxysql.cfg /etc/proxysql.cnf
CMD ["/usr/bin/proxysql", "--idle-threads", "-c", "/etc/proxysql.cnf", "-f"]proxysql.cfgadmin_variables=
{
admin_credentials="admin:admin;cluster1:clusterpass"
cluster_username="cluster1"
cluster_password="clusterpass"
cluster_check_interval_ms=200
cluster_check_status_frequency=100
}
proxysql_servers =
(
{
hostname="proxysql1"
port=6032
weight=100
comment="proxysql1"
},
{
hostname="proxysql2"
port=6032
weight=100
comment="proxysql2"
}
)docker-compose up --build --force-recreatefirst console docker-compose exec proxysql1 bash -c "mysql -h 127.0.0.1 -P 6032 -u admin -padmin -e \"insert into mysql_servers VALUES(1,'127.0.0.1','3306',0,'OFFLINE_SOFT',0,0,0,0,1,0,'bar'); LOAD MYSQL SERVERS TO RUNTIME;\"; while true; do mysql -h 127.0.0.1 -P 6032 -u admin -padmin -e \"update mysql_servers set comment='\${RANDOM}' where hostname='127.0.0.1'; load mysql servers to runtime;\"; echo \"saved \${RANDOM}\"; sleep 1; done"second console docker-compose exec proxysql2 bash -c "while true; do timeout 5 mysql -h 127.0.0.1 -P 6032 -u admin -padmin -e 'select weight,hostname,port,comment from runtime_proxysql_servers order by weight' >/dev/null 2>&1 || break ; echo \"saved \${RANDOM}\"; done; echo 'stucked';"Wait 1 minute |
|
Hi @f4rx . Do you mind trying to reproduce it branch Note: branch |
|
The v2.1.2-cluster branch also leads to a hang. However, I catch an error on a working insolation, the example with docker composite does not repeat it. I can send the core file to the mail. Thread 5 (Thread 0x7f08f787e700 (LWP 21378)):
#0 0x00007f08fd2a1474 in read () from target:/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x000055b48c3e948e in sock_read ()
#2 0x000055b48c3e7e0a in bread_conv ()
#3 0x000055b48c3e6991 in BIO_read ()
#4 0x000055b48c385ab3 in ssl3_read_n ()
#5 0x000055b48c38a78d in ssl3_get_record ()
#6 0x000055b48c387d82 in ssl3_read_bytes ()
#7 0x000055b48c38eaa0 in ssl3_read ()
#8 0x000055b48c397e92 in ssl_read_internal ()
#9 0x000055b48c3980a3 in SSL_read ()
#10 0x000055b48c68c7fe in ma_tls_read (ctls=0x7f08f6cde2a0, buffer=buffer@entry=0x7f08f6cdf740 "\001", length=length@entry=16384) at /opt/proxysql/deps/mariadb-client-library/mariadb_client/libmariadb/secure/openssl.c:729
#11 0x000055b48c67fdf5 in ma_pvio_tls_read (ctls=<optimized out>, buffer=buffer@entry=0x7f08f6cdf740 "\001", length=length@entry=16384) at /opt/proxysql/deps/mariadb-client-library/mariadb_client/libmariadb/ma_tls.c:90
#12 0x000055b48c67f5a6 in ma_pvio_read (pvio=pvio@entry=0x7f08f6c25240, buffer=0x7f08f6cdf740 "\001", length=length@entry=16384) at /opt/proxysql/deps/mariadb-client-library/mariadb_client/libmariadb/ma_pvio.c:250
#13 0x000055b48c67f753 in ma_pvio_cache_read (pvio=0x7f08f6c25240, buffer=buffer@entry=0x7f08f6c16000 "\001\001", length=length@entry=4) at /opt/proxysql/deps/mariadb-client-library/mariadb_client/libmariadb/ma_pvio.c:297
#14 0x000055b48c698f86 in ma_real_read (net=0x7f08f6c01500, complen=complen@entry=0x7f08f787b378) at /opt/proxysql/deps/mariadb-client-library/mariadb_client/libmariadb/ma_net.c:371
#15 0x000055b48c6999ed in ma_net_read (net=net@entry=0x7f08f6c01500) at /opt/proxysql/deps/mariadb-client-library/mariadb_client/libmariadb/ma_net.c:425
#16 0x000055b48c67b0bc in ma_net_safe_read (mysql=mysql@entry=0x7f08f6c01500, is_data_packet=is_data_packet@entry=0x0) at /opt/proxysql/deps/mariadb-client-library/mariadb_client/libmariadb/mariadb_lib.c:205
#17 0x000055b48c67ed0a in mthd_my_read_query_result (mysql=0x7f08f6c01500) at /opt/proxysql/deps/mariadb-client-library/mariadb_client/libmariadb/mariadb_lib.c:2536
#18 0x000055b48c298fac in ProxySQL_Cluster::fetch_and_store (this=0x7f08fa659a80, conn=0x7f08f6c01500, f_query=..., result=0x7f08f787b620) at ProxySQL_Cluster.cpp:1142
#19 0x000055b48c29a017 in ProxySQL_Cluster::pull_mysql_servers_from_peer (this=0x7f08fa659a80) at ProxySQL_Cluster.cpp:1282
#20 0x000055b48c290215 in ProxySQL_Node_Entry::set_checksums (this=0x7f08fa6acfc0, _r=0x7f08f6db2900) at ProxySQL_Cluster.cpp:634
#21 0x000055b48c29f173 in ProxySQL_Cluster_Nodes::Update_Node_Checksums (this=0x7f08fa659ac0, _h=0x7f08fa62c630 "10.250.0.25", _p=6032, _r=0x7f08f6db2900) at ProxySQL_Cluster.cpp:1902
#22 0x000055b48c2a6272 in ProxySQL_Cluster::Update_Node_Checksums (this=0x7f08fa659a80, _h=0x7f08fa62c630 "10.250.0.25", _p=6032, _r=0x7f08f6db2900) at ../include/ProxySQL_Cluster.hpp:290
#23 0x000055b48c28d378 in ProxySQL_Cluster_Monitor_thread (args=0x7f08f85b7c00) at ProxySQL_Cluster.cpp:173
#24 0x00007f08fd2976db in start_thread () from target:/lib/x86_64-linux-gnu/libpthread.so.0
#25 0x00007f08fc27571f in clone () from target:/lib/x86_64-linux-gnu/libc.so.6
Thread 30 (Thread 0x7f08edece700 (LWP 30502)):
#0 0x00007f08fd2a11fd in __lll_lock_wait () from target:/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f08fd29a025 in pthread_mutex_lock () from target:/lib/x86_64-linux-gnu/libpthread.so.0
#2 0x000055b48c1f31a4 in ProxySQL_Admin::mysql_servers_wrlock (this=0x7f08fa65a500) at ProxySQL_Admin.cpp:5459
#3 0x000055b48c1e91b7 in ProxySQL_Admin::GenericRefreshStatistics (this=0x7f08fa65a500, query_no_space=0x7f08f4446da0 "SELECT writer_hostgroup, backup_writer_hostgroup, reader_hostgroup, offline_hostgroup, active, max_writers, writer_is_also_reader, max_transactions_behind, comment FROM runtime_mysql_group_replication"..., query_no_space_length=211,
admin=true) at ProxySQL_Admin.cpp:3252
#4 0x000055b48c1eb641 in admin_session_handler (sess=0x7f08f4547600, _pa=0x7f08fa65a500, pkt=0x7f08edecbf60) at ProxySQL_Admin.cpp:3892
#5 0x000055b48c196247 in MySQL_Session::handler___status_WAITING_CLIENT_DATA___STATE_SLEEP___MYSQL_COM_QUERY___not_mysql (this=0x7f08f4547600, pkt=...) at MySQL_Session.cpp:3197
#6 0x000055b48c197f23 in MySQL_Session::get_pkts_from_client (this=0x7f08f4547600, wrong_pass=@0x7f08edecbee3: false, pkt=...) at MySQL_Session.cpp:3666
#7 0x000055b48c19a569 in MySQL_Session::handler (this=0x7f08f4547600) at MySQL_Session.cpp:4250
#8 0x000055b48c1f1654 in child_mysql (arg=0x7f08f3c0cab0) at ProxySQL_Admin.cpp:5027
#9 0x00007f08fd2976db in start_thread () from target:/lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f08fc27571f in clone () from target:/lib/x86_64-linux-gnu/libc.so.6 |
|
@renecannao Any updates? |
|
Closing because this is likely to be already fixed in a lot of rework done on Cluster in the last years |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
ProxySQL version 2.0.15-20-g32bb92cd_DEBUG, codename Trul (3 Nodes)
Ubuntu 18.04.6 LTS
I have a deadlock of two threads
pt-pmp:
thread 30 sets lock for mysql_servers_lock and waits
mutexthread 5 sets lock for
mutexand waits mysql_servers_lockWe couldn’t describe steps needed to replay the problem. It occurs occaisionally on 3 Nodes cluster when updating settings. https://github.com/percona/pxc_scheduler_handler is used as a scheduler.
The text was updated successfully, but these errors were encountered: