ThreadSanitizer: lock-order-inversion (potential deadlock)

[ilejn]

Description

Trying to switch ClickHouse https://github.com/ClickHouse/ClickHouse to recent librdkafka
Codebase https://github.com/ilejn/librdkafka/commits/branch_2.4.0 contains #4724 .
The issue

E           Exception: Sanitizer assert found for instance ==================
E           WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=1)
E             Cycle in lock order graph: M0 (0x722c005a18f0) => M1 (0x722c005a1580) => M0
E           
E             Mutex M1 acquired here while holding mutex M0 in thread T706:
E               #0 pthread_mutex_lock <null> (clickhouse+0x72a3aca) (BuildId: 1a2f287f5febb9d6f32c5da82f5b1a8b42479177)
E               #1 mtx_lock build_docker/./contrib/librdkafka/src/tinycthread.c:136:10 (clickhouse+0x1d2313f9) (BuildId: 1a2f287f5febb9d6f32c5da82f5b1a8b42479177)
E               #2 rd_kafka_q_keep build_docker/./contrib/librdkafka/src/rdkafka_queue.h:152:9 (clickhouse+0x1d1bea91) (BuildId: 1a2f287f5febb9d6f32c5da82f5b1a8b42479177)
E               #3 rd_kafka_q_fwd_set0 build_docker/./contrib/librdkafka/src/rdkafka_queue.c:182:17 (clickhouse+0x1d1bea91)
E               #4 rd_kafka_toppar_op_fetch_start build_docker/./contrib/librdkafka/src/rdkafka_partition.c:2278:17 (clickhouse+0x1d1b7ba5) (BuildId: 1a2f287f5febb9d6f32c5da82f5b1a8b42479177)
E               #5 rd_kafka_assignment_serve_pending build_docker/./contrib/librdkafka/src/rdkafka_assignment.c:511:25 (clickhouse+0x1d114cbf) (BuildId: 1a2f287f5febb9d6f32c5da82f5b1a8b42479177)
E               #6 rd_kafka_assignment_serve build_docker/./contrib/librdkafka/src/rdkafka_assignment.c:632:31 (clickhouse+0x1d114cbf)
E               #7 rd_kafka_assignment_apply_offsets build_docker/./contrib/librdkafka/src/rdkafka_assignment.c:238:17 (clickhouse+0x1d116937) (BuildId: 1a2f287f5febb9d6f32c5da82f5b1a8b42479177)
E               #8 rd_kafka_assignment_handle_OffsetFetch build_docker/./contrib/librdkafka/src/rdkafka_assignment.c:330:9 (clickhouse+0x1d116937)
E               #9 rd_kafka_buf_callback build_docker/./contrib/librdkafka/src/rdkafka_buf.c:509:17 (clickhouse+0x1d102194) (BuildId: 1a2f287f5febb9d6f32c5da82f5b1a8b42479177)
E               #10 rd_kafka_buf_handle_op build_docker/./contrib/librdkafka/src/rdkafka_buf.c:452:9 (clickhouse+0x1d102fba) (BuildId: 1a2f287f5febb9d6f32c5da82f5b1a8b42479177)
E               #11 rd_kafka_op_handle_std build_docker/./contrib/librdkafka/src/rdkafka_op.c:875:17 (clickhouse+0x1d1af19d) (BuildId: 1a2f287f5febb9d6f32c5da82f5b1a8b42479177)
E               #12 rd_kafka_op_handle build_docker/./contrib/librdkafka/src/rdkafka_op.c:915:15 (clickhouse+0x1d1af19d)
E               #13 rd_kafka_q_serve build_docker/./contrib/librdkafka/src/rdkafka_queue.c:578:23 (clickhouse+0x1d1c0b9a) (BuildId: 1a2f287f5febb9d6f32c5da82f5b1a8b42479177)
E               #14 rd_kafka_thread_main build_docker/./contrib/librdkafka/src/rdkafka.c:2136:17 (clickhouse+0x1d1077f0) (BuildId: 1a2f287f5febb9d6f32c5da82f5b1a8b42479177)
E               #15 _thrd_wrapper_function build_docker/./contrib/librdkafka/src/tinycthread.c:576:9 (clickhouse+0x1d2317bb) (BuildId: 1a2f287f5febb9d6f32c5da82f5b1a8b42479177)
E           
E               Hint: use TSAN_OPTIONS=second_deadlock_stack=1 to get more informative warning message
E           
E             Mutex M0 acquired here while holding mutex M1 in thread T2:
E               #0 pthread_mutex_lock <null> (clickhouse+0x72a3aca) (BuildId: 1a2f287f5febb9d6f32c5da82f5b1a8b42479177)
E               #1 mtx_lock build_docker/./contrib/librdkafka/src/tinycthread.c:136:10 (clickhouse+0x1d2313f9) (BuildId: 1a2f287f5febb9d6f32c5da82f5b1a8b42479177)
E               #2 rd_kafka_q_disable0 build_docker/./contrib/librdkafka/src/rdkafka_queue.h:194:17 (clickhouse+0x1d1b171d) (BuildId: 1a2f287f5febb9d6f32c5da82f5b1a8b42479177)
E               #3 rd_kafka_q_destroy0 build_docker/./contrib/librdkafka/src/rdkafka_queue.h:222:17 (clickhouse+0x1d1b171d)
E               #4 rd_kafka_q_destroy_owner build_docker/./contrib/librdkafka/src/rdkafka_queue.h:248:9 (clickhouse+0x1d1b171d)
E               #5 rd_kafka_toppar_destroy_final build_docker/./contrib/librdkafka/src/rdkafka_partition.c:341:9 (clickhouse+0x1d1b171d)
E               #6 rd_kafka_op_destroy build_docker/./contrib/librdkafka/src/rdkafka_op.c:480:9 (clickhouse+0x1d1addf6) (BuildId: 1a2f287f5febb9d6f32c5da82f5b1a8b42479177)
E               #7 rd_kafka_op_filter build_docker/./contrib/librdkafka/src/rdkafka_queue.c:363:17 (clickhouse+0x1d1c034b) (BuildId: 1a2f287f5febb9d6f32c5da82f5b1a8b42479177)
E               #8 rd_kafka_q_pop_serve build_docker/./contrib/librdkafka/src/rdkafka_queue.c:421:40 (clickhouse+0x1d1c034b)
E               #9 rd_kafka_q_pop build_docker/./contrib/librdkafka/src/rdkafka_queue.c:497:16 (clickhouse+0x1d1c07b2) (BuildId: 1a2f287f5febb9d6f32c5da82f5b1a8b42479177)
E               #10 rd_kafka_consume0 build_docker/./contrib/librdkafka/src/rdkafka.c:3212:19 (clickhouse+0x1d108517) (BuildId: 1a2f287f5febb9d6f32c5da82f5b1a8b42479177)
E               #11 rd_kafka_consumer_poll build_docker/./contrib/librdkafka/src/rdkafka.c:3314:16 (clickhouse+0x1d1089c6) (BuildId: 1a2f287f5febb9d6f32c5da82f5b1a8b42479177)
E               #12 cppkafka::Consumer::poll(std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l>>) build_docker/./contrib/cppkafka/src/consumer.cpp:266:12 (clickhouse+0x1d0e27ae) (BuildId: 1a2f287f5febb9d6f32c5da82f5b1a8b42479177)
E               #13 DB::KafkaConsumer::drain() build_docker/./src/Storages/Kafka/KafkaConsumer.cpp:207:30 (clickhouse+0x15d4c044) (BuildId: 1a2f287f5febb9d6f32c5da82f5b1a8b42479177)
E               #14 DB::KafkaConsumer::~KafkaConsumer() build_docker/./src/Storages/Kafka/KafkaConsumer.cpp:187:13 (clickhouse+0x15d4cfab) (BuildId: 1a2f287f5febb9d6f32c5da82f5b1a8b42479177)

Looking into this, while any assistance is appreciated.

How to reproduce

I've failed to reproduce the issue not using ClickHouse.
In principle it is just deleting a topic that is happening simultaneously with unsubscription.

More details are https://s3.amazonaws.com/clickhouse-test-reports/65314/dfba81e5f7172e729974479dec58e8dc40c75de1/integration_tests__tsan__[2_6]//home/ubuntu/actions-runner/_work/_temp/test/output_dir/integration_run_parallel1_0.log

Checklist

IMPORTANT: We will close issues where the checklist has not been completed.

Please provide the following information:

• librdkafka version (release number or git tag): <2.4.0 with patches>
• Apache Kafka version: <confluentinc/cp-kafka:5.2.0>
• librdkafka client configuration: <auto.reset.offset=earliest>
• Operating system: <ubuntu 22.04>