TiFlash may crash when stopping, "resource manager grpc call failed: AcquireTokenBuckets"

[JaySon-Huang]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

2. What did you expect to see? (Required)

3. What did you see instead (Required)

[2024/03/21 04:12:43.057 +08:00] [ERROR] [BaseDaemon.cpp:358] [\"(from thread 1) Terminate called after throwing an instance of pingcap::Exception\\nCode: 2, e.displayText() = Exception: resource manager grpc call failed: AcquireTokenBuckets. write failed: failed to connect to all addresses, e.what() = Exception\\nStack trace:\\n\\n\\n       0x77d4b1d\\tterminate_handler() [tiflash+125651741]\\n                \\tlibs/libdaemon/src/BaseDaemon.cpp:624\\n  0x7ff0b084da13\\tstd::__terminate(void (*)()) [libc++abi.so.1+236051]\\n  0x7ff0b084d9b8\\tstd::terminate() [libc++abi.so.1+235960]\\n       0x1ee372b\\t__clang_call_terminate [tiflash+32388907]\\n       0x1fbbc4b\\tDB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) [tiflash+33274955]\\n                \\tdbms/src/Server/Server.cpp:1833\\n       0x922e59a\\tPoco::Util::Application::run() [tiflash+153281946]\\n                \\tcontrib/poco/Util/src/Application.cpp:335\\n       0x1fa\"] [source=BaseDaemon] [thread_id=970]

4. What is your TiFlash version? (Required)

master

[JaySon-Huang]

Should be caused by the exception thrown in LocalAdmissionController::stop

tiflash/dbms/src/Flash/ResourceControl/LocalAdmissionController.h

Line 572 in 3234fef

fetchTokensFromGAC(acquire_infos, "before stop", true);

[JaySon-Huang]

compile error

>  ninja gtests_dbms tiflash -j32                                                                                                                                                                               2024-04-12 18:03:07
[0/2] Re-checking globbed directories...
[41/44] Building CXX object dbms/src/Server/CMakeFiles/tiflash-server-lib.dir/Server.cpp.o
FAILED: dbms/src/Server/CMakeFiles/tiflash-server-lib.dir/Server.cpp.o 
/DATA/disk1/ra_common/tiflash-env-15/sysroot/bin/ccache /DATA/disk1/ra_common/tiflash-env-15/sysroot/bin/clang++ -DAWS_SDK_VERSION_MAJOR=1 -DAWS_SDK_VERSION_MINOR=11 -DAWS_SDK_VERSION_PATCH=186 -DBOOST_BIND_GLOBAL_PLACEHOLDERS -DBOOST_NO_CXX98_FUNCTION_BASE -DBOOST_SYSTEM_NO_DEPRECATED -DCARES_STATICLIB -DDBMS_PUBLIC_GTEST -DFIU_ENABLE -DLZ4_DISABLE_DEPRECATE_WARNINGS=1 -DLZ4_FAST_DEC_LOOP=1 -DMULTIPLE_CONTEXT_GTEST -DPOCO_STATIC -DPOCO_UNBUNDLED_ZLIB -DSIMDJSON_THREADS_ENABLED=1 -DSTACK_LINE_READER_BUFFER_SIZE=1024 -DTIFLASH_COMPILER_VPCLMULQDQ_SUPPORT=1 -DTIFLASH_ENABLE_AVX512_SUPPORT=1 -DTIFLASH_ENABLE_AVX_SUPPORT=1 -DTIFLASH_SOURCE_PREFIX=\"/DATA/disk1/jaysonhuang/tiflash-master\" -D_LIBCPP_ENABLE_THREAD_SAFETY_ANNOTATIONS -I/DATA/disk1/jaysonhuang/tiflash-master/dbms/src -I/DATA/disk1/jaysonhuang/tiflash-master/libs/libdaemon/include -I/DATA/disk1/jaysonhuang/tiflash-master/contrib/GmSSL/include -I/DATA/disk1/jaysonhuang/tiflash-master/contrib/double-conversion -I/DATA/disk1/jaysonhuang/tiflash-master/contrib/boost -I/DATA/disk1/jaysonhuang/tiflash-master/contrib/xxHash -I/DATA/disk1/jaysonhuang/tiflash-master/cmake-build-debug/dbms/src -I/DATA/disk1/jaysonhuang/tiflash-master/contrib/libpcg-random/include -I/DATA/disk1/jaysonhuang/tiflash-master/contrib/libcityhash/include -I/DATA/disk1/jaysonhuang/tiflash-master/libs/libcommon/include -I/DATA/disk1/jaysonhuang/tiflash-master/cmake-build-debug/libs/libcommon/include -I/DATA/disk1/jaysonhuang/tiflash-master/libs/libpocoext/include -I/DATA/disk1/jaysonhuang/tiflash-master/contrib/poco/Data/include -I/DATA/disk1/jaysonhuang/tiflash-master/contrib/poco/Foundation/include -I/DATA/disk1/jaysonhuang/tiflash-master/contrib/zlib-ng -I/DATA/disk1/jaysonhuang/tiflash-master/cmake-build-debug/contrib/zlib-ng -I/DATA/disk1/jaysonhuang/tiflash-master/contrib/poco/Util/include -I/DATA/disk1/jaysonhuang/tiflash-master/contrib/poco/XML/include -I/DATA/disk1/jaysonhuang/tiflash-master/contrib/poco/JSON/include -I/DATA/disk1/jaysonhuang/tiflash-master/contrib/poco/Net/include -I/DATA/disk1/jaysonhuang/tiflash-master/contrib/cctz/include -I/DATA/disk1/jaysonhuang/tiflash-master/cmake-build-debug/contrib/jemalloc-cmake/include -I/DATA/disk1/jaysonhuang/tiflash-master/cmake-build-debug/contrib/jemalloc-cmake/include/jemalloc -I/DATA/disk1/jaysonhuang/tiflash-master/contrib/jemalloc/include -I/DATA/disk1/jaysonhuang/tiflash-master/contrib/jemalloc-cmake/include -I/DATA/disk1/jaysonhuang/tiflash-master/contrib/cpu_features/include -I/DATA/disk1/jaysonhuang/tiflash-master/contrib/libcpuid/include -I/DATA/disk1/jaysonhuang/tiflash-master/contrib/lz4/lib -I/DATA/disk1/jaysonhuang/tiflash-master/contrib/zstd/lib -I/DATA/disk1/jaysonhuang/tiflash-master/contrib/client-c/third_party/libfiu/libfiu -I/DATA/disk1/jaysonhuang/tiflash-master/contrib/prometheus-cpp/core/include -I/DATA/disk1/jaysonhuang/tiflash-master/cmake-build-debug/contrib/prometheus-cpp-cmake/core/include -I/DATA/disk1/jaysonhuang/tiflash-master/contrib/prometheus-cpp/push/include -I/DATA/disk1/jaysonhuang/tiflash-master/cmake-build-debug/contrib/prometheus-cpp-cmake/push/include -I/DATA/disk1/jaysonhuang/tiflash-master/contrib/prometheus-cpp/pull/include -I/DATA/disk1/jaysonhuang/tiflash-master/cmake-build-debug/contrib/prometheus-cpp-cmake/pull/include -I/DATA/disk1/jaysonhuang/tiflash-master/contrib/cpptoml -I/DATA/disk1/jaysonhuang/tiflash-master/contrib/magic_enum/include -I/DATA/disk1/jaysonhuang/tiflash-master/contrib/simdjson/include -I/DATA/disk1/jaysonhuang/tiflash-master/contrib/poco/NetSSL_OpenSSL/include -I/DATA/disk1/jaysonhuang/tiflash-master/contrib/poco/Crypto/include -I/DATA/disk1/jaysonhuang/tiflash-master/contrib/xxHash/cmake_unofficial/.. -I/DATA/disk1/jaysonhuang/tiflash-master/contrib/tiflash-proxy/raftstore-proxy/ffi/src -I/DATA/disk1/jaysonhuang/tiflash-master/contrib/libbtrie/include -I/DATA/disk1/jaysonhuang/tiflash-master/contrib/abseil-cpp -I/DATA/disk1/jaysonhuang/tiflash-master/cmake-build-debug/contrib/etcd -I/DATA/disk1/jaysonhuang/tiflash-master/cmake-build-debug/contrib/etcd/etcd -I/DATA/disk1/jaysonhuang/tiflash-master/contrib/grpc/include -I/DATA/disk1/jaysonhuang/tiflash-master/cmake-build-debug/contrib/kvproto/cpp -I/DATA/disk1/jaysonhuang/tiflash-master/cmake-build-debug/contrib/kvproto/cpp/kvproto -I/DATA/disk1/jaysonhuang/tiflash-master/contrib/client-c/include -I/DATA/disk1/jaysonhuang/tiflash-master/cmake-build-debug/contrib/grpc/third_party/cares/cares -I/DATA/disk1/jaysonhuang/tiflash-master/contrib/grpc/third_party/cares/cares -I/DATA/disk1/jaysonhuang/tiflash-master/cmake-build-debug/contrib/tipb/cpp -I/DATA/disk1/jaysonhuang/tiflash-master/contrib/libunwind/include -I/DATA/disk1/jaysonhuang/tiflash-master/contrib/libdivide -I/DATA/disk1/jaysonhuang/tiflash-master/contrib/libmetrohash/src -I/DATA/disk1/jaysonhuang/tiflash-master/contrib/libfarmhash -isystem /DATA/disk1/jaysonhuang/tiflash-master/libs/libprocess_metrics/include -isystem /DATA/disk1/jaysonhuang/tiflash-master/contrib/fmtlib-cmake/../fmtlib/include -isystem /DATA/disk1/jaysonhuang/tiflash-master/libs/libsymbolization/include -isystem /DATA/disk1/jaysonhuang/tiflash-master/contrib/re2 -isystem /DATA/disk1/jaysonhuang/tiflash-master/cmake-build-debug/contrib/re2-cmake -isystem /DATA/disk1/jaysonhuang/tiflash-master/contrib/boringssl/include -isystem /DATA/disk1/jaysonhuang/tiflash-master/contrib/aws/src/aws-cpp-sdk-core/include -isystem /DATA/disk1/jaysonhuang/tiflash-master/cmake-build-debug/contrib/aws-cmake/include -isystem /DATA/disk1/jaysonhuang/tiflash-master/contrib/aws/generated/src/aws-cpp-sdk-s3/include -isystem /DATA/disk1/jaysonhuang/tiflash-master/contrib/aws/generated/src/aws-cpp-sdk-sts/include -isystem /DATA/disk1/jaysonhuang/tiflash-master/contrib/aws-c-auth/include -isystem /DATA/disk1/jaysonhuang/tiflash-master/contrib/aws-c-common/include -isystem /DATA/disk1/jaysonhuang/tiflash-master/contrib/aws-c-io/include -isystem /DATA/disk1/jaysonhuang/tiflash-master/contrib/aws-crt-cpp/include -isystem /DATA/disk1/jaysonhuang/tiflash-master/contrib/aws-c-mqtt/include -isystem /DATA/disk1/jaysonhuang/tiflash-master/contrib/aws-c-sdkutils/include -isystem /DATA/disk1/jaysonhuang/tiflash-master/contrib/protobuf/src -Wthread-safety  -pipe -mssse3 -msse4.1 -msse4.2 -mpclmul -mpopcnt  -fno-omit-frame-pointer  -Wall -Wno-unused-command-line-argument  -Wnon-virtual-dtor  -stdlib=libc++ -Wextra -Werror -g -g3 -ggdb3 -O0 -fverbose-asm -fno-inline  -D_LIBCPP_DEBUG=0 -fPIC -DDUMMY_BACKTRACE -std=gnu++20 -MD -MT dbms/src/Server/CMakeFiles/tiflash-server-lib.dir/Server.cpp.o -MF dbms/src/Server/CMakeFiles/tiflash-server-lib.dir/Server.cpp.o.d -o dbms/src/Server/CMakeFiles/tiflash-server-lib.dir/Server.cpp.o -c /DATA/disk1/jaysonhuang/tiflash-master/dbms/src/Server/Server.cpp
/DATA/disk1/jaysonhuang/tiflash-master/dbms/src/Server/Server.cpp:1789:60: error: no member named 'safeStop' in 'DB::MockLocalAdmissionController'
                LocalAdmissionController::global_instance->safeStop();
                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
/DATA/disk1/jaysonhuang/tiflash-master/libs/libcommon/include/ext/scope_guard.h:46:50: note: expanded from macro 'SCOPE_EXIT'
#define SCOPE_EXIT(...) SCOPE_EXIT_FWD(__LINE__, __VA_ARGS__)
                                                 ^~~~~~~~~~~
/DATA/disk1/jaysonhuang/tiflash-master/libs/libcommon/include/ext/scope_guard.h:45:53: note: expanded from macro 'SCOPE_EXIT_FWD'
#define SCOPE_EXIT_FWD(n, ...) SCOPE_EXIT_CONCAT(n, __VA_ARGS__)
                                                    ^~~~~~~~~~~
/DATA/disk1/jaysonhuang/tiflash-master/libs/libcommon/include/ext/scope_guard.h:44:90: note: expanded from macro 'SCOPE_EXIT_CONCAT'
#define SCOPE_EXIT_CONCAT(n, ...) const auto scope_exit##n = ext::make_scope_guard([&] { __VA_ARGS__; })
                                                                                         ^~~~~~~~~~~
/DATA/disk1/jaysonhuang/tiflash-master/dbms/src/Server/Server.cpp:1812:60: error: no member named 'safeStop' in 'DB::MockLocalAdmissionController'
                LocalAdmissionController::global_instance->safeStop();
                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
/DATA/disk1/jaysonhuang/tiflash-master/libs/libcommon/include/ext/scope_guard.h:46:50: note: expanded from macro 'SCOPE_EXIT'
#define SCOPE_EXIT(...) SCOPE_EXIT_FWD(__LINE__, __VA_ARGS__)
                                                 ^~~~~~~~~~~
/DATA/disk1/jaysonhuang/tiflash-master/libs/libcommon/include/ext/scope_guard.h:45:53: note: expanded from macro 'SCOPE_EXIT_FWD'
#define SCOPE_EXIT_FWD(n, ...) SCOPE_EXIT_CONCAT(n, __VA_ARGS__)
                                                    ^~~~~~~~~~~
/DATA/disk1/jaysonhuang/tiflash-master/libs/libcommon/include/ext/scope_guard.h:44:90: note: expanded from macro 'SCOPE_EXIT_CONCAT'
#define SCOPE_EXIT_CONCAT(n, ...) const auto scope_exit##n = ext::make_scope_guard([&] { __VA_ARGS__; })