Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Short lived connections causing segmentation fault #4310

Closed
dsabeti opened this issue Aug 30, 2018 · 7 comments · Fixed by #4323
Closed

Short lived connections causing segmentation fault #4310

dsabeti opened this issue Aug 30, 2018 · 7 comments · Fixed by #4323
Assignees
@ggreenway
Labels
bug

Comments

@dsabeti
Copy link

dsabeti commented Aug 30, 2018

Description:
When a client attempts to make a very short connection -- for example, checking that a port is open -- Envoy crashes with a seg fault. For context, we have health check logic that tries to establish these short connections periodically.

Repro steps:

  1. Start with envoyproxoy/envoy:3b47cbabb517db69a40f38843a405d1d6f742f5d docker image.
  2. Run Envoy (see config below):
envoy -c /config/config.yaml --service-cluster c --service-node n &
  1. Run a listener "app":
nc -l -p 8080 &
  1. Attempt a short connection to the listener via Envoy:
nc -z localhost 61001
  1. Observe the Segfault:
[2018-08-30 21:15:19.120][138][debug][filter] source/common/tcp_proxy/tcp_proxy.cc:157] [C0] new tcp proxy session
[2018-08-30 21:15:19.120][138][debug][filter] source/common/tcp_proxy/tcp_proxy.cc:293] [C0] Creating connection to cluster 0-service-cluster
[2018-08-30 21:15:19.120][138][debug][pool] source/common/tcp/conn_pool.cc:67] creating a new connection
[2018-08-30 21:15:19.120][138][debug][pool] source/common/tcp/conn_pool.cc:325] [C1] connecting
[2018-08-30 21:15:19.120][138][debug][connection] source/common/network/connection_impl.cc:576] [C1] connecting to 10.255.139.5:8080
[2018-08-30 21:15:19.121][138][debug][connection] source/common/network/connection_impl.cc:585] [C1] connection in progress
[2018-08-30 21:15:19.121][138][debug][pool] source/common/tcp/conn_pool.cc:92] queueing request due to no available connections
[2018-08-30 21:15:19.121][138][debug][main] source/server/connection_handler_impl.cc:218] [C0] new connection
[2018-08-30 21:15:19.121][138][debug][connection] source/common/ssl/ssl_socket.cc:131] [C0] handshake error: 5
[2018-08-30 21:15:19.121][138][debug][connection] source/common/network/connection_impl.cc:133] [C0] closing socket: 0
[2018-08-30 21:15:19.122][138][debug][main] source/server/connection_handler_impl.cc:51] [C0] adding to cleanup list
[2018-08-30 21:15:19.122][138][debug][connection] source/common/network/connection_impl.cc:466] [C1] connected
[2018-08-30 21:15:19.122][138][debug][pool] source/common/tcp/conn_pool.cc:249] [C1] assigning connection
[2018-08-30 21:15:19.122][138][debug][filter] source/common/tcp_proxy/tcp_proxy.cc:467] TCP:onUpstreamEvent(), requestedServerName:
[2018-08-30 21:15:19.122][138][debug][connection] source/common/network/connection_impl.cc:98] [C1] closing data_to_write=0 type=1
[2018-08-30 21:15:19.123][138][debug][connection] source/common/network/connection_impl.cc:133] [C1] closing socket: 1
[2018-08-30 21:15:19.123][138][debug][pool] source/common/tcp/conn_pool.cc:107] [C1] client disconnected
[2018-08-30 21:15:19.123][138][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:125] Caught Segmentation fault, suspect faulting address 0x80

Config:
config.yml:

admin:
  access_log_path: /dev/null
  address:
    socket_address:
      address: 127.0.0.1
      port_value: 61003
static_resources:
  clusters:
  - name: 0-service-cluster
    connect_timeout: 0.25s
    type: STATIC
    lb_policy: ROUND_ROBIN
    hosts:
    - socket_address:
        address: 127.0.0.1
        port_value: 8080
    circuit_breakers:
      thresholds:
      - max_connections: 4294967295
dynamic_resources:
  lds_config:
    path: /config/listeners.yaml

listeners.yml:

version_info: "0"
resources:
- '@type': type.googleapis.com/envoy.api.v2.Listener
  name: listener-8080
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 61001
  filter_chains:
  - filters:
    - name: envoy.tcp_proxy
      config:
        stat_prefix: 0-stats
        cluster: 0-service-cluster
    tls_context:
      common_tls_context:
        tls_certificates:
        - certificate_chain:
            inline_string: |
              -----BEGIN CERTIFICATE-----
              MIID9jCCAt6gAwIBAgIQBz3mQy9iS35PA89k8+AnDzANBgkqhkiG9w0BAQsFADBD
              MQwwCgYDVQQGEwNVU0ExFjAUBgNVBAoTDUNsb3VkIEZvdW5kcnkxGzAZBgNVBAMT
              Emluc3RhbmNlSWRlbnRpdHlDQTAeFw0xODA4MzAyMTA1NDRaFw0xODA4MzEyMTA1
              NDRaMIHIMYGeMDgGA1UECxMxb3JnYW5pemF0aW9uOmE0OTJlZjIxLTRjY2YtNDU3
              Ny05NzhlLWFiNDc1OTEzNGUxNjAxBgNVBAsTKnNwYWNlOjBiY2ZlNTA3LWRmYmMt
              NGRhMy1iNGUzLTNjZGZlMjFjY2FhMzAvBgNVBAsTKGFwcDo5YmVkMTI5Mi1kOThl
              LTRmMDYtOWMzYS01MTE2MjNkZWJmYzMxJTAjBgNVBAMTHDBiNGQxZDhkLTZkN2It
              NGUzMi02MDRkLTViN2QwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCu
              ocWoukTrzeSiEacAiO9j7exjKb+Wf5LW6hCpi7pY/U9NdQvYG70R1eVYzvxu1Rhg
              2KergFDmEK8prqIfuG4WD1Je0nBk5f7jXXj+4Gpmke1ZRRrq4Ie3TxVDFv5xRc9W
              UB2SygQvDH6DxIy3ipsbyNLftGyABEmuLCDLP4ttshKw6Aq5BO/mmTglGbANFu05
              6ki1s000bevGaC9r+q/28Ro6gNiW60tnSv/WjriaT8R/UYjyuqaoBvJMYZNlPB58
              nRDaGOj67+XTtttRFhzgwA8bAsdSn9o7N1gl9rs/lDDjV2X0ZbycnWeys67TWl+T
              A0kDcfWnTa/3BIqqbH1pAgMBAAGjYDBeMA4GA1UdDwEB/wQEAwIDqDAdBgNVHSUE
              FjAUBggrBgEFBQcDAgYIKwYBBQUHAwEwLQYDVR0RBCYwJIIcMGI0ZDFkOGQtNmQ3
              Yi00ZTMyLTYwNGQtNWI3ZIcECv+LPzANBgkqhkiG9w0BAQsFAAOCAQEAfcgYjtwE
              DSuqBtH9SD2bKhZVW8qKCSuWCu5VC9JN1ilWjNQaIuksvVRCeAPCFKmzJ/7odvQ4
              ERoJY2taddM3nURnaLPacKrBUuZ6g3d9NPEZa5vUImojDWS/wcO06iAVNNcostOJ
              +4nrk3VZFiHZL0tAQGpwZC9G7Pz2oOBD8/sDmncaPByKBCP8lo3Yb5yz05eoKAiI
              1dpqKbeboLDNnmLmmvwpho+prW+ezGo5tGhWltcwHH/HDmOa0+udq7OPTLiKN9d6
              f8M2YbaksGNcygN+PWlKFlUZD22m/8KpXk5DQqYACsI6dfU84gB+7EJViziwsBjz
              ZIdT9YI21DqFYg==
              -----END CERTIFICATE-----
              -----BEGIN CERTIFICATE-----
              MIIDKzCCAhOgAwIBAgIRAJnmAqJ7LL0zLTWpRzA6d/swDQYJKoZIhvcNAQELBQAw
              OjEMMAoGA1UEBhMDVVNBMRYwFAYDVQQKEw1DbG91ZCBGb3VuZHJ5MRIwEAYDVQQD
              EwlhcHBSb290Q0EwHhcNMTgwODMwMTcwOTQxWhcNMTkwODMwMTcwOTQxWjBDMQww
              CgYDVQQGEwNVU0ExFjAUBgNVBAoTDUNsb3VkIEZvdW5kcnkxGzAZBgNVBAMTEmlu
              c3RhbmNlSWRlbnRpdHlDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
              AJhDW9kdTlSRKosFo4iVFZRGoeg3oRgd+xp10j1qoEUPHLF6+mhqhsjV2XqnCf4C
              7ULyDKZvseVK0qzo0OfLfVX6d+mDdnry/veatxD71sIHf7Dq8Z3z1goawSn91HBL
              6Ogzx4fxN/WMoMmwUF9DFq4goAoI0uUdRqN7PLvowucbqMAlTW1M3agbfVRGUmo2
              Qk4C5en4J9XkupQaROP2ZG7ktbsMj215C+f/w+pcwrm5uqkYkIkTZGSm4nol7xuB
              xNN63X/rnTnulEecCKO5fdOIf6yU5ijCNjSQ3teCM2HBje/JMUSmc+yj3eHewQaY
              XmqmT2Mkb3IRUdTsoMP6JG0CAwEAAaMjMCEwDgYDVR0PAQH/BAQDAgEGMA8GA1Ud
              EwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBANDHj3ON0WCPGGwXg33vT9ll
              LlijXd7ek8cEMp/Axjnqv8NoBZZ89DJfVPWyatz44txMELezFUKL8x4kQLRdkS9l
              7lSzx6r3NeVr4HRERFsFCMeT2++MIm0POyleaY+odjDN3PLGdzPLnFe3S/+iFOa7
              Dtii5bG0aB83NebVLmKzVKi1OlZxUVIMHBxWPqblH9UPF+UB7syCAGxtYXujd6cl
              fUly0QVowxPPQ13yst038fWkfKtNVf/paXt1dxvAgce9O923WIgGOTpSFfxLNuno
              uX7ewXOHplteJsZxyV1U44Y1RThmu7NLRdDFT32cQPo45HdjAnKdxV/UEDGuidc=
              -----END CERTIFICATE-----
          private_key:
            inline_string: |
              -----BEGIN RSA PRIVATE KEY-----
              MIIEpAIBAAKCAQEArqHFqLpE683kohGnAIjvY+3sYym/ln+S1uoQqYu6WP1PTXUL
              2Bu9EdXlWM78btUYYNinq4BQ5hCvKa6iH7huFg9SXtJwZOX+4114/uBqZpHtWUUa
              6uCHt08VQxb+cUXPVlAdksoELwx+g8SMt4qbG8jS37RsgARJriwgyz+LbbISsOgK
              uQTv5pk4JRmwDRbtOepItbNNNG3rxmgva/qv9vEaOoDYlutLZ0r/1o64mk/Ef1GI
              8rqmqAbyTGGTZTwefJ0Q2hjo+u/l07bbURYc4MAPGwLHUp/aOzdYJfa7P5Qw41dl
              9GW8nJ1nsrOu01pfkwNJA3H1p02v9wSKqmx9aQIDAQABAoIBAAkL4lRqSdW5VcW7
              il4xuc0Ab0V3zS/7rdyqRXCzB1VC0g/eJ13rmfDJltbx6gvHyDqEHrN4szgn1xse
              q9Fe2X0VEBxPZHklnP2kQvdZIsnrd1MvjERsmzs4wo42K7osdqAvCEkpY+wYBpnG
              3c9IQ969mYbzLse28SYvrQc6ZHurLvCyKFQooETXJsPRyJLkK3fPcaca/m+dc8jV
              qIJBSJIMTVwDI7zPKehn1tlXsUQLxjfoPKHFV+6xJyhU4rGRWAxJ+Ud53tfbBqpv
              tLSFCRFh1nmmdhkBOVdKPtskUW+0WGm9/sSi1sLWQu8imJtWd0EVYSdUvzARvXAi
              50AkH30CgYEA4Gm0gJJHKuqQyCEr7hMpusKDOo8HWDmAvFGxJrxh/XZeYZZEQJwU
              6AfNsJ6nSgpuFGcPRb4obVdlRcWjcWKK1dSgRIYTJpikpC/93rZpvknr3gKuMazx
              a8NA/HuKFCW3mj6dEU3aFAQdf1qq5BFc5W5xmVEFh7Tujse+/rQsCT8CgYEAxzZN
              oL+90AirV5YZArLDeJg12zYUuqVs93vYVkjCfBw3+nJH7FylK4fGW5IyyoAeccJD
              Rwzo9kb5J7Oa/gIWIRikFDsNWeSt2Mqt0O180e1mo5kzzWlolJWz/ZtBQFKB3guf
              +dC6ALF0/bqfxbXrfHBg9PDKfu0/36rCY2QDZ1cCgYATAgEO7tTmVq3itaSR6r+b
              GQSifiDiISS9TrIRlN076b2+yEqJqm6+5IJQlm+W/uEqCzyo7J1h/YgPa9kXLoCR
              8guNIkoW99DiM0puhcbZHkvDJzj37i0YspYbjJ86a6Nshicpow14qoyRD+3vmWTj
              7gHjOZCdWP7nyxVrX3lnYwKBgQCTNYilyisMyOs83Pj8andR7UCXDd7Bki70rvrN
              wBSfvAaNttd5TXjssvecgWuxAn382G342gxouUey0+Er5QTPTjMpk/2ouPjSo+tP
              4b8ELidTq4mBOGyEqB+Kv6i3TIEVQYflBh39FyV5vegFJQqwOSccubcZll5F/RyK
              b8SoHwKBgQDb2LGEiNecwQ+iTSRGNn8FbvNzP/PFb98p0Dzy6+7iVuqGuBSFhLgT
              w+LtNeN5zhioSqCTxEQLQ4HxYXm4U+HJ0SubnWSWFAFoHqU9IA3KirzEudycq6CW
              XXc4s05QAXI1e9JJ222q8PShhkyT5ruYt9aeOSslSvMaqB3W2Fuybg==
              -----END RSA PRIVATE KEY-----
        tls_params:
          cipher_suites: '[ECDHE-RSA-AES256-GCM-SHA384|ECDHE-RSA-AES128-GCM-SHA256]'
        validation_context:
          trusted_ca:
            inline_string: |
              -----BEGIN CERTIFICATE-----
              MIIDJDCCAgygAwIBAgIRALXtOp85ebqiIifjBw5nScswDQYJKoZIhvcNAQELBQAw
              OzEMMAoGA1UEBhMDVVNBMRYwFAYDVQQKEw1DbG91ZCBGb3VuZHJ5MRMwEQYDVQQD
              EwppbnRlcm5hbENBMB4XDTE4MDgzMDE3MDkyN1oXDTE5MDgzMDE3MDkyN1owOzEM
              MAoGA1UEBhMDVVNBMRYwFAYDVQQKEw1DbG91ZCBGb3VuZHJ5MRMwEQYDVQQDEwpp
              bnRlcm5hbENBMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAzfAurTTy
              l99unSzDJ5JorUZBDqdbuEwZqNYMbyWLUVpthXwMlWxmj9f+KOrIvCXDlTaq0zaf
              u4ZLzX6UtZYtWjQnPTVRGQzOMrO2Alu6/G5hEn6wBv3gjYTD5HXv2N8OAdCbMRm7
              4FQfTyF5pdTacEOmWtkqMAvTU0yQSgorXxCRchZmp2T4W7/1w/A3VqyZTzWBgVAL
              Dmt76Pi3harkY0eecgSRO6i3CCLI0DI3ZgjF/4W8rB6d23ejBbGqsIPA7KJV2hOm
              fvvXBI9sTu/mCgQgQ/4uGtWI7XDCV1aeXd6YCAW3vwoIJF+vu09pDSAmBmRSxHO7
              Pt13sph3UXknPQIDAQABoyMwITAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUw
              AwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAn/ux7CVPH0Vwj/IQLr6r6avQna7179Tk
              6gDoO7NWD67sggVKsWfofHKerI8HuN9aHhZjnOUHtWC+3SGJPXQNcOYBjGTMQTN6
              13hBb+CGf+Lpj6QPcdXUZhcAmsxhuR4Oq0HHs4819v32AaZT4rcUhBbsr3C8Pkap
              iyviaRwkv9vvARrotb/rivRQcNt35ujIWUZMhg/jGtfWIzcQfgOElgg0Lzaqm+Zv
              rj+WGPTZkur6CMAKh/eALMuVA8CqcghBjbJijOcAJujbMQKXzDhgVXFLT/a/ZADI
              81j5Nbzk/KM2r2NqIPeHT2Qi3J9H93ayf9obN+GcpcZV8BzHM6bs7Q==
              -----END CERTIFICATE-----
          verify_subject_alt_name:
          - gorouter.service.cf.internal
      require_client_certificate: true

Call Stack:

#0  0x0000000000ce6c99 in uw_frame_state_for ()
#1  0x0000000000ce88d8 in _Unwind_Backtrace ()
#2  0x0000000000c57cac in backward::details::Unwinder<backward::StackTraceImpl<backward::system_tag::linux_tag>::callback>::operator() (depth=72, f=..., this=0x7fa11c8647c0) at external/com_github_bombela_backward/backward.hpp:647
#3  backward::details::unwind<backward::StackTraceImpl<backward::system_tag::linux_tag>::callback> (depth=72, f=...) at external/com_github_bombela_backward/backward.hpp:688
#4  backward::StackTraceImpl<backward::system_tag::linux_tag>::load_here (this=this@entry=0x7fa11c864820, depth=depth@entry=72) at external/com_github_bombela_backward/backward.hpp:704
#5  0x0000000000c59265 in backward::StackTraceImpl<backward::system_tag::linux_tag>::load_from (depth=64, addr=0x80, this=0x7fa11c864820) at external/com_github_bombela_backward/backward.hpp:710
#6  Envoy::BackwardsTrace::captureFrom (address=0x80, this=0x7fa11c864820) at bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:66
#7  Envoy::SignalAction::sigHandler (sig=11, info=<optimized out>, context=<optimized out>) at source/exe/signal_action.cc:33 
#8  <signal handler called>
#9  0x0000000000000080 in ?? ()
#10 0x00000000006f054a in Envoy::Network::ConnectionImpl::state (this=0x31e3a40) at source/common/network/connection_impl.cc:119
#11 0x00000000008c1efe in Envoy::TcpProxy::Filter::onUpstreamEvent (this=0x319ea10, event=Envoy::Network::ConnectionEvent::LocalClose) at source/common/tcp_proxy/tcp_proxy.cc:452
#12 0x00000000006f3f85 in Envoy::Network::ConnectionImpl::raiseEvent (this=0x31e3b80, event=Envoy::Network::ConnectionEvent::LocalClose) at source/common/network/connection_impl.cc:279
#13 0x00000000006f1acd in Envoy::Network::ConnectionImpl::closeSocket (this=0x31e3b80, close_type=Envoy::Network::ConnectionEvent::LocalClose) at source/common/network/connection_impl.cc:144
#14 0x00000000006f1c06 in Envoy::Network::ConnectionImpl::close (this=0x31e3b80, type=Envoy::Network::ConnectionCloseType::NoFlush) at source/common/network/connection_impl.cc:107
#15 0x00000000008c3250 in Envoy::TcpProxy::Filter::~Filter (this=0x319ea10, __in_chrg=<optimized out>) at source/common/tcp_proxy/tcp_proxy.cc:143
#16 0x00000000006f50e1 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x319ea00) at /usr/include/c++/5/bits/shared_ptr_base.h:150
#17 std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=0x31f1268, __in_chrg=<optimized out>) at /usr/include/c++/5/bits/shared_ptr_base.h:659
#18 std::__shared_ptr<Envoy::Network::ReadFilter, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (this=0x31f1260, __in_chrg=<optimized out>) at /usr/include/c++/5/bits/shared_ptr_base.h:925
#19 std::shared_ptr<Envoy::Network::ReadFilter>::~shared_ptr (this=0x31f1260, __in_chrg=<optimized out>) at /usr/include/c++/5/bits/shared_ptr.h:93
#20 Envoy::Network::FilterManagerImpl::ActiveReadFilter::~ActiveReadFilter (this=0x31f1240, __in_chrg=<optimized out>)
    at bazel-out/k8-opt/bin/source/common/network/_virtual_includes/filter_manager_lib/common/network/filter_manager_impl.h:52
#21 Envoy::Network::FilterManagerImpl::ActiveReadFilter::~ActiveReadFilter (this=0x31f1240, __in_chrg=<optimized out>)
    at bazel-out/k8-opt/bin/source/common/network/_virtual_includes/filter_manager_lib/common/network/filter_manager_impl.h:52
#22 0x00000000006f21d2 in std::default_delete<Envoy::Network::FilterManagerImpl::ActiveReadFilter>::operator() (this=<optimized out>, __ptr=<optimized out>) at /usr/include/c++/5/bits/unique_ptr.h:76
#23 std::unique_ptr<Envoy::Network::FilterManagerImpl::ActiveReadFilter, std::default_delete<Envoy::Network::FilterManagerImpl::ActiveReadFilter> >::~unique_ptr (this=0x31eaa10, __in_chrg=<optimized out>)
    at /usr/include/c++/5/bits/unique_ptr.h:236
#24 std::_List_node<std::unique_ptr<Envoy::Network::FilterManagerImpl::ActiveReadFilter, std::default_delete<Envoy::Network::FilterManagerImpl::ActiveReadFilter> > >::~_List_node (this=0x31eaa00, __in_chrg=<optimized out>)
    at /usr/include/c++/5/bits/stl_list.h:106
#25 __gnu_cxx::new_allocator<std::_List_node<std::unique_ptr<Envoy::Network::FilterManagerImpl::ActiveReadFilter, std::default_delete<Envoy::Network::FilterManagerImpl::ActiveReadFilter> > > >::destroy<std::_List_node<std::unique_ptr<Envoy::Network::FilterManagerImpl::Act
iveReadFilter, std::default_delete<Envoy::Network::FilterManagerImpl::ActiveReadFilter> > > > (this=<optimized out>, __p=0x31eaa00) at /usr/include/c++/5/ext/new_allocator.h:124
#26 std::__cxx11::_List_base<std::unique_ptr<Envoy::Network::FilterManagerImpl::ActiveReadFilter, std::default_delete<Envoy::Network::FilterManagerImpl::ActiveReadFilter> >, std::allocator<std::unique_ptr<Envoy::Network::FilterManagerImpl::ActiveReadFilter, std::default_d
elete<Envoy::Network::FilterManagerImpl::ActiveReadFilter> > > >::_M_clear (this=0x31e3a78) at /usr/include/c++/5/bits/list.tcc:75
#27 std::__cxx11::_List_base<std::unique_ptr<Envoy::Network::FilterManagerImpl::ActiveReadFilter, std::default_delete<Envoy::Network::FilterManagerImpl::ActiveReadFilter> >, std::allocator<std::unique_ptr<Envoy::Network::FilterManagerImpl::ActiveReadFilter, std::default_d
elete<Envoy::Network::FilterManagerImpl::ActiveReadFilter> > > >::~_List_base (this=0x31e3a78, __in_chrg=<optimized out>) at /usr/include/c++/5/bits/stl_list.h:446
#28 std::__cxx11::list<std::unique_ptr<Envoy::Network::FilterManagerImpl::ActiveReadFilter, std::default_delete<Envoy::Network::FilterManagerImpl::ActiveReadFilter> >, std::allocator<std::unique_ptr<Envoy::Network::FilterManagerImpl::ActiveReadFilter, std::default_delete<
Envoy::Network::FilterManagerImpl::ActiveReadFilter> > > >::~list (this=0x31e3a78, __in_chrg=<optimized out>) at /usr/include/c++/5/bits/stl_list.h:507
#29 Envoy::Network::FilterManagerImpl::~FilterManagerImpl (this=0x31e3a58, __in_chrg=<optimized out>) at bazel-out/k8-opt/bin/source/common/network/_virtual_includes/filter_manager_lib/common/network/filter_manager_impl.h:39
#30 Envoy::Network::ConnectionImpl::~ConnectionImpl (this=this@entry=0x31e3a40, __in_chrg=<optimized out>, __vtt_parm=<optimized out>) at source/common/network/connection_impl.cc:70
#31 0x00000000006f22a1 in Envoy::Network::ConnectionImpl::~ConnectionImpl (this=0x31e3a40, __in_chrg=<optimized out>, __vtt_parm=<optimized out>) at source/common/network/connection_impl.cc:78
#32 0x00000000006e8e71 in Envoy::Server::ConnectionHandlerImpl::ActiveConnection::~ActiveConnection (this=0x31f1500, __in_chrg=<optimized out>) at source/server/connection_handler_impl.cc:244
#33 0x00000000006ece0d in std::default_delete<Envoy::Event::DeferredDeletable>::operator() (this=<optimized out>, __ptr=<optimized out>) at /usr/include/c++/5/bits/unique_ptr.h:76
#34 std::unique_ptr<Envoy::Event::DeferredDeletable, std::default_delete<Envoy::Event::DeferredDeletable> >::reset (__p=<optimized out>, this=<optimized out>) at /usr/include/c++/5/bits/unique_ptr.h:344
#35 Envoy::Event::DispatcherImpl::clearDeferredDeleteList (this=0x3134b00) at source/common/event/dispatcher_impl.cc:68
#36 0x0000000000a01418 in event_process_active_single_queue (base=base@entry=0x3136840, max_to_process=max_to_process@entry=2147483647, endtime=endtime@entry=0x0, activeq=<optimized out>) at ../event.c:1646
#37 0x0000000000a01b0f in event_process_active (base=base@entry=0x3136840) at ../event.c:1738
#38 0x0000000000a04898 in event_base_loop (base=0x3136840, flags=<optimized out>) at ../event.c:1961
#39 0x00000000006e8986 in Envoy::Server::WorkerImpl::threadRoutine (this=0x312cf30, guard_dog=...) at source/server/worker_impl.cc:94
#40 0x0000000000a52c6e in Envoy::Thread::Thread::Thread(std::function<void ()>)::{lambda(void*)#1}::_FUN(void*) () at /usr/include/c++/5/functional:2267
#41 0x00007fa134a0a6ba in start_thread (arg=0x7fa11c877700) at pthread_create.c:333
#42 0x00007fa13443741d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

cc @jvshahid
@ggreenway
Copy link
Contributor

@PiotrSikora I'm guessing this is the same issue as istio/istio#8310.

@ggreenway
Copy link
Contributor

Nevermind; not quite the same I don't think.

@ggreenway ggreenway added the bug label Aug 31, 2018
@PiotrSikora
Copy link
Contributor

It doesn't look like it is.

Also, it seems that this started very recently (unless I'm misreading this), whereas TCP proxy changes have been merged ~3 weeks ago.

@PiotrSikora
Copy link
Contributor

Actually, it's the same... I got confused by the dbg (with working ASSERT()s) vs opt stacktrace and new log entries that have been added since istio's envoy.

@PiotrSikora
Copy link
Contributor

Yeah, I verified that it started happening with 04616d6.

@zuercher do you have a quick fix in mind or should we revert it?

@PiotrSikora
Copy link
Contributor

Also, it's worth noting that this is only happening for TLS listeners.

Ultimately, I think that the proper fix for this is marking connection as "connected" and connecting to upstream only after the TLS handshake completed successfully (see the discussion in #2800).

@ggreenway
Copy link
Contributor

I agree that not connecting to an upstream until after the TLS handshake completes would improve the behavior.

However, my guess is that this can also happen with non-TLS connections, it just depends on the timing. I think it's a sequence like, or similar to:

  • downstream connection established
  • upstream connection initiated
  • downstream connection disconnected
  • upstream connection established

Another possibility is that handshake failure interacts incorrectly with half-close semantics, and the handshake failure is considered a half-close.

@ggreenway ggreenway self-assigned this Aug 31, 2018
ggreenway added a commit to ggreenway/envoy that referenced this issue Aug 31, 2018
@ggreenway
Fix crash in tcp_proxy.
b64f67c 
Closing the upstream connection is not safe from the Filter destructor,
because it triggers events back into the downstream connection, which
is partially destructed.

Ensure that the upstream connection is closed before the destructor is called.

Fixes envoyproxy#4310

Signed-off-by: Greg Greenway <[email protected]>
@ggreenway ggreenway mentioned this issue Aug 31, 2018
Merged
zuercher added a commit to turbinelabs/envoy that referenced this issue Sep 5, 2018
@zuercher
tcp_proxy: reproduce envoyproxy#4310 in a test
6c88504 
Signed-off-by: Stephan Zuercher <[email protected]>
@ggreenway ggreenway mentioned this issue Sep 5, 2018
Closed
ggreenway added a commit that referenced this issue Sep 5, 2018
@ggreenway
Fix crash in tcp_proxy (#4323)
e34dcd6 
* Fix crash in tcp_proxy.

Closing the upstream connection is not safe from the Filter destructor,
because it triggers events back into the downstream connection, which
is partially destructed.

Ensure that the upstream connection is closed before the destructor is called.

Fixes #4310

Signed-off-by: Greg Greenway <[email protected]>
@jvshahid jvshahid mentioned this issue Sep 12, 2018
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ggreenway ggreenway

Labels
bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix crash in tcp_proxy ggreenway/envoy
3 participants
@PiotrSikora @dsabeti @ggreenway