Ambassador 0.53.1 auth spans missing/disconnected

[flands]

Describe the bug

After upgrading to Ambassador 0.53.1 from 0.52.0 auth spans are no longer seen in traces

To Reproduce

Ambassador 0.52.0 trace example:

Ambassador 0.53.1. trace example:

Also note in Ambassador 0.53.1 auth login requests are hanging traces (not connected to Ambassador):

To repro just upgrade from 0.52.0 to 0.53.1.

Expected behavior

Same behavior as 0.52.0 -- auth calls should be part of traces

Versions (please complete the following information):

• Ambassador: [e.g. 0.32.1] 0.53.1
• Kubernetes environment [e.g. Minikube, bare metal, Google Kubernetes Engine] kops
• Version [e.g. 1.8.1] 1.11.7

Additional context
Add any other context about the problem here.

[kflynn]

Closing #1572 as a dup of this one.

[gsagula]

@flands Can you try this image, please? quay.io/gsagula/ambassador:master-030d897c

Thank you!

[richarddli]

Fixed in 0.72

[sergeevo]

Hi, I'm facing the same issue in Ambassador 1.4.1, Envoy doesn't send tracing headers to ext auth service:

2020-06-05 11:06:00.829][429][debug][http] [source/common/http/conn_manager_impl.cc:259] [C5261] new stream
[2020-06-05 11:06:00.829][429][debug][http] [source/common/http/conn_manager_impl.cc:708] [C5261][S3120272034051349865] request headers complete (end_stream=true):
':authority', 'gateway-test.com'
':path', '/test-path/'
':method', 'GET'
'accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.9'
'accept-encoding', 'gzip, deflate, br'
'accept-language', 'en-US,en;q=0.9,ru-RU;q=0.8,ru;q=0.7'
'cache-control', 'max-age=0'
'cookie', 'SESSION_ID=N2M0YmRlOTktNTIwZC00NjJiLWE4YzItMjc3N2Y0YjQzNjc2'
'sec-fetch-dest', 'document'
'sec-fetch-mode', 'navigate'
'sec-fetch-site', 'cross-site'
'sec-fetch-user', '?1'
'upgrade-insecure-requests', '1'
'user-agent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.122 Safari/537.36'
'x-forwarded-for', '1.1.1.1'
'x-forwarded-port', '443'
'x-forwarded-proto', 'https'
'connection', 'keep-alive'
[2020-06-05 11:06:00.829][429][debug][http] [source/common/http/conn_manager_impl.cc:1257] [C5261][S3120272034051349865]
request end stream
[2020-06-05 11:06:00.829][429][debug][lua] [source/extensions/filters/common/lua/lua.cc:37] coroutine finished
[2020-06-05 11:06:00.829][429][debug][router] [source/common/router/router.cc:434] [C0][S10785466150304944309] cluster 'cluster_extauth_localhost_8080_gateway_test' match for URL '/test-path/'
[2020-06-05 11:06:00.830][429][debug][router] [source/common/router/router.cc:549] [C0][S10785466150304944309] router decoding headers:
':authority', 'gateway-test.com'
':method', 'GET'
':path', '/test-path/'
':scheme', 'http'
'content-length', '0'
'x-forwarded-proto', 'https'
'cookie', 'SESSION_ID=N2M0YmRlOTktNTIwZC00NjJiLWE4YzItMjc3N2Y0YjQzNjc2'
'x-request-id', '83b73502-8c37-94e3-a43d-12608f2996cb'
'x-forwarded-for', '0.0.0.0,0.0.0.0'
'user-agent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.122 Safari/537.36'
'x-envoy-internal', 'true'
'x-envoy-expected-rq-timeout-ms', '5000'
[2020-06-05 11:06:00.830][429][debug][pool] [source/common/http/http1/conn_pool.cc:104] [C5245] using existing connection
[2020-06-05 11:06:00.830][429][debug][router] [source/common/router/router.cc:1618] [C0][S10785466150304944309] pool ready
[2020-06-05 11:06:00.830][431][debug][conn_handler] [source/server/connection_handler_impl.cc:335] [C5322] new connection
[2020-06-05 11:06:00.845][429][debug][client] [source/common/http/codec_client.cc:101] [C5245] response complete
[2020-06-05 11:06:00.845][429][debug][router] [source/common/router/router.cc:1036] [C0][S10785466150304944309] upstream
headers complete: end_stream=true
[2020-06-05 11:06:00.845][429][debug][http] [source/common/http/async_client_impl.cc:93] async http request response headers (end_stream=true):
':status', '200'
'uri', '/**'
'authorization', 'Bearer token',
'x-content-type-options', 'nosniff'
'x-xss-protection', '1; mode=block'
'cache-control', 'no-cache, no-store, max-age=0, must-revalidate'
'pragma', 'no-cache'
'expires', '0'
'strict-transport-security', 'max-age=31536000 ; includeSubDomains'
'x-frame-options', 'DENY'
'content-length', '0'
'date', 'Fri, 05 Jun 2020 11:06:00 GMT'
'x-envoy-upstream-service-time', '15'
[2020-06-05 11:06:00.845][429][debug][filter] [source/extensions/filters/http/ext_authz/ext_authz.cc:163] [C5261][S3120272034051349865] ext_authz has cleared route cache

That is my ext_auth config:

apiVersion: getambassador.io/v2
kind: AuthService
name: authentication
ambassador_id: gateway-test
auth_service: localhost:8080
status_on_error:
code: 500
allowed_request_headers:
- "x-b3-traceid"
- "x-b3-spanid"
- "x-b3-parentspanid"
- "x-b3-sampled"

Is it Ambassador or Envoy issue?