//test/integration:api_listener_integration_test flakes #9746

lizan · 2020-01-20T08:59:55Z

bazel test //test/integration:api_listener_integration_test --runs_per_test=1000 results:

//test/integration:api_listener_integration_test                         FAILED in 81 out of 1000 in 7.6s

Call Stack:

exec ${PAGER:-/usr/bin/less} "$0" || exit 1
Executing tests from //test/integration:api_listener_integration_test
-----------------------------------------------------------------------------
WARNING: Perftools heap leak checker is active -- Performance may suffer
[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from IpVersions/ApiListenerIntegrationTest
[ RUN      ] IpVersions/ApiListenerIntegrationTest.Basic/IPv4
[2020-01-20 08:56:55.998][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:83] Caught Segmentation fault, suspect faulting address 0x8
[2020-01-20 08:56:55.998][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:70] Backtrace (use tools/stack_decode.py to get line numbers):
[2020-01-20 08:56:55.998][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:71] Envoy version: 0/1.13.0-dev/redacted/DEBUG/BoringSSL
[2020-01-20 08:56:56.013][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #0: Envoy::SignalAction::sigHandler() [0x2d5c43c]
[2020-01-20 08:56:56.013][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #1: __restore_rt [0x7f7880acb390]
[2020-01-20 08:56:56.027][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #2: Envoy::Http::TlsCachingDateProviderImpl::setDateHeader() [0x2592054]
[2020-01-20 08:56:56.040][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #3: Envoy::Http::ConnectionManagerImpl::ActiveStream::encodeHeaders() [0x256df5a]
[2020-01-20 08:56:56.054][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #4: Envoy::Http::ConnectionManagerImpl::ActiveStream::sendLocalReply()::$_6::operator()() [0x2574d10]
[2020-01-20 08:56:56.067][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #5: std::_Function_handler<>::_M_invoke() [0x2574a8c]
[2020-01-20 08:56:56.081][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #6: std::function<>::operator()() [0x2af8802]
[2020-01-20 08:56:56.094][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #7: Envoy::Http::Utility::sendLocalReply() [0x2af401c]
[2020-01-20 08:56:56.107][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #8: Envoy::Http::ConnectionManagerImpl::ActiveStream::sendLocalReply() [0x2564fd1]
[2020-01-20 08:56:56.120][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #9: Envoy::Http::ConnectionManagerImpl::ActiveStreamDecoderFilter::sendLocalReply() [0x257b875]
[2020-01-20 08:56:56.133][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #10: Envoy::Router::Filter::onUpstreamAbort() [0x27107c8]
[2020-01-20 08:56:56.146][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #11: Envoy::Router::Filter::onUpstreamReset() [0x2710fb0]
[2020-01-20 08:56:56.159][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #12: Envoy::Router::Filter::UpstreamRequest::onResetStream() [0x2715d87]
[2020-01-20 08:56:56.172][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #13: Envoy::Http::StreamCallbackHelper::runResetCallbacks() [0x268cddf]
[2020-01-20 08:56:56.185][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #14: Envoy::Http::Http1::ClientConnectionImpl::onResetStream() [0x268ad06]
[2020-01-20 08:56:56.198][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #15: Envoy::Http::Http1::ConnectionImpl::onResetStreamBase() [0x2683f3b]
[2020-01-20 08:56:56.211][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #16: Envoy::Http::Http1::StreamEncoderImpl::resetStream() [0x2683dbf]
[2020-01-20 08:56:56.224][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #17: Envoy::Http::CodecClient::onEvent() [0x267cba0]
[2020-01-20 08:56:56.237][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #18: Envoy::Network::ConnectionImplBase::raiseConnectionEvent() [0x29bfb31]
[2020-01-20 08:56:56.251][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #19: Envoy::Network::ConnectionImpl::raiseEvent() [0x29b6e15]
[2020-01-20 08:56:56.264][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #20: Envoy::Network::ConnectionImpl::closeSocket() [0x29b598d]
[2020-01-20 08:56:56.277][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #21: Envoy::Network::ConnectionImpl::closeConnectionImmediately() [0x29b57ea]
[2020-01-20 08:56:56.291][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #22: Envoy::Network::ConnectionImpl::close() [0x29b52e4]
[2020-01-20 08:56:56.304][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #23: Envoy::Http::CodecClient::close() [0x267c51a]
[2020-01-20 08:56:56.317][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #24: Envoy::Http::Http1::ConnPoolImpl::~ConnPoolImpl() [0x1c3f587]
[2020-01-20 08:56:56.331][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #25: Envoy::Http::Http1::ProdConnPoolImpl::~ProdConnPoolImpl() [0x1c44668]
[2020-01-20 08:56:56.344][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #26: Envoy::Http::Http1::ProdConnPoolImpl::~ProdConnPoolImpl() [0x1c4468c]
[2020-01-20 08:56:56.357][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #27: std::default_delete<>::operator()() [0x19abb1f]
[2020-01-20 08:56:56.370][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #28: std::unique_ptr<>::~unique_ptr() [0x19aba63]
[2020-01-20 08:56:56.383][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #29: std::pair<>::~pair() [0x19ab9f6]
[2020-01-20 08:56:56.396][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #30: __gnu_cxx::new_allocator<>::destroy<>() [0x19ab9c9]
[2020-01-20 08:56:56.409][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #31: absl::allocator_traits<>::destroy_impl<>() [0x19ab9a3]
[2020-01-20 08:56:56.422][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #32: absl::allocator_traits<>::destroy<>() [0x19ab971]
[2020-01-20 08:56:56.435][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #33: absl::container_internal::map_slot_policy<>::destroy<>() [0x19ab940]
[2020-01-20 08:56:56.448][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #34: absl::container_internal::FlatHashMapPolicy<>::destroy<>() [0x19ab90d]
[2020-01-20 08:56:56.462][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #35: absl::container_internal::hash_policy_traits<>::destroy<>() [0x19ab47d]
[2020-01-20 08:56:56.475][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #36: absl::container_internal::raw_hash_set<>::clear() [0x19ab27e]
[2020-01-20 08:56:56.488][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #37: Envoy::Upstream::ConnPoolMap<>::clearActivePools() [0x19ab128]
[2020-01-20 08:56:56.501][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #38: Envoy::Upstream::ConnPoolMap<>::~ConnPoolMap() [0x19ab01c]
[2020-01-20 08:56:56.514][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #39: std::default_delete<>::operator()() [0x19aaedb]
[2020-01-20 08:56:56.527][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #40: std::unique_ptr<>::~unique_ptr() [0x19ac4b3]
[2020-01-20 08:56:56.540][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #41: std::array<>::~array() [0x19aaca8]
[2020-01-20 08:56:56.552][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #42: Envoy::Upstream::PriorityConnPoolMap<>::~PriorityConnPoolMap() [0x19ac5e5]
[2020-01-20 08:56:56.565][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #43: __gnu_cxx::new_allocator<>::destroy<>() [0x19ac5c9]
[2020-01-20 08:56:56.578][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #44: std::allocator_traits<>::destroy<>() [0x19ac560]
[2020-01-20 08:56:56.591][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #45: std::_Sp_counted_ptr_inplace<>::_M_dispose() [0x19aa88c]
[2020-01-20 08:56:56.605][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #46: std::_Sp_counted_base<>::_M_release() [0x165ba5c]
[2020-01-20 08:56:56.618][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #47: std::__shared_count<>::~__shared_count() [0x165ba0a]
[2020-01-20 08:56:56.631][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #48: std::__shared_ptr<>::~__shared_ptr() [0x19a9efe]
[2020-01-20 08:56:56.644][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #49: std::shared_ptr<>::~shared_ptr() [0x19a2c08]
[2020-01-20 08:56:56.657][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #50: Envoy::Upstream::ClusterManagerImpl::ThreadLocalClusterManagerImpl::ConnPoolsContainer::~ConnPoolsContainer() [0x19a3535]
[2020-01-20 08:56:56.670][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #51: std::pair<>::~pair() [0x19a9506]
[2020-01-20 08:56:56.683][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #52: __gnu_cxx::new_allocator<>::destroy<>() [0x19a94d9]
[2020-01-20 08:56:56.696][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #53: std::allocator_traits<>::destroy<>() [0x19a93e0]
[2020-01-20 08:56:56.709][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #54: std::__detail::_Hashtable_alloc<>::_M_deallocate_node() [0x19a92f6]
[2020-01-20 08:56:56.722][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #55: std::__detail::_Hashtable_alloc<>::_M_deallocate_nodes() [0x19a9255]
[2020-01-20 08:56:56.736][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #56: std::_Hashtable<>::clear() [0x19a9179]
[2020-01-20 08:56:56.749][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #57: std::unordered_map<>::clear() [0x19a25d5]
[2020-01-20 08:56:56.763][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #58: Envoy::Upstream::ClusterManagerImpl::ThreadLocalClusterManagerImpl::~ThreadLocalClusterManagerImpl() [0x1994e12]
[2020-01-20 08:56:56.777][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #59: __gnu_cxx::new_allocator<>::destroy<>() [0x19b6ebc]
[2020-01-20 08:56:56.790][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #60: std::allocator_traits<>::destroy<>() [0x19b6e50]
[2020-01-20 08:56:56.803][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #61: std::_Sp_counted_ptr_inplace<>::_M_dispose() [0x19b6c3c]
[2020-01-20 08:56:56.803][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #62: std::_Sp_counted_base<>::_M_release() [0x165ba5c]
[2020-01-20 08:56:56.803][19][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #63: std::__shared_count<>::~__shared_count() [0x165ba0a]
ActiveStream 0x4dded00, stream_id_: 190603338098846370, has_continue_headers_: 0, is_head_request_: 0, decoding_headers_only_: 0, encoding_headers_only_: 0
request_headers_: 
  ':method', 'GET'
  ':path', '/api'
  ':scheme', 'http'
  ':authority', 'host'
  'x-forwarded-proto', 'http'
  'x-request-id', 'd0790074-f639-49f7-8210-8aee287f4220'
  'x-envoy-expected-rq-timeout-ms', '15000'
request_trailers_: null
response_headers_: 
  ':status', '503'
  'content-length', '95'
  'content-type', 'text/plain'
response_trailers_: null
&stream_info_: 
  StreamInfoImpl 0x4ddee38, protocol_: 1, response_code_: null, response_code_details_: upstream_reset_before_response_started{connection termination}, health_check_request_: 0, route_name_: 
external/bazel_tools/tools/test/test-setup.sh: line 310:    17 Segmentation fault      (core dumped) "${TEST_PATH}" "$@" 2>&1

The text was updated successfully, but these errors were encountered:

junr03 · 2020-01-21T18:27:32Z

Thanks for reporting @lizan! I'll look at it today

Description: previously the test was not waiting for the expectation on the server's thread to complete. Therefore, there was a use after free race condition with the ApiListener's TlsCachingDateProvider. This PR makes it so that the test waits for the expectation to be fulfilled and thus prevents the race. Risk Level: low Testing: ran integration test a few thousand times locally on a linux machine. Fixes #9746 Signed-off-by: Jose Nino <[email protected]>

…9959) Description: this PR adds a shutdown method to the ApiListener and calls it where appropriate during server termination. Previously there would be a crash due to use after free of objects in thread local storage by streams in the ApiListener. Funny enough the flakes reported in #9746 happened due to this. Risk Level: low Testing: new unit and integration test. Without appropriate termination the new integration test repros the stacktrace reported in #9746. Signed-off-by: Jose Nino <[email protected]>

Description: previously the test was not waiting for the expectation on the server's thread to complete. Therefore, there was a use after free race condition with the ApiListener's TlsCachingDateProvider. This PR makes it so that the test waits for the expectation to be fulfilled and thus prevents the race. Risk Level: low Testing: ran integration test a few thousand times locally on a linux machine. Fixes envoyproxy#9746 Signed-off-by: Jose Nino <[email protected]> Signed-off-by: Antonio Vicente <[email protected]>

lizan added the area/test flakes label Jan 20, 2020

lizan assigned junr03 Jan 20, 2020

junr03 mentioned this issue Jan 23, 2020

test flake: fix flake in ApiListener integration test #9808

Merged

lizan closed this as completed in #9808 Jan 24, 2020

junr03 mentioned this issue Feb 7, 2020

api listener: add shutdown method and call during server termination #9959

Merged

antoniovicente mentioned this issue Sep 29, 2020

backport to v1.13: test flake: fix flake in ApiListener integration test (#9808) #13316

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

//test/integration:api_listener_integration_test flakes #9746

//test/integration:api_listener_integration_test flakes #9746

lizan commented Jan 20, 2020

junr03 commented Jan 21, 2020

//test/integration:api_listener_integration_test flakes #9746

//test/integration:api_listener_integration_test flakes #9746

Comments

lizan commented Jan 20, 2020

junr03 commented Jan 21, 2020