[HTTP] Body Buffering Not Working on Per-Route ExtAuthZ Filter

[agrawroh]

Description

I'm trying to use the with_request_body on per-route ExtAuthZ filter but it's not working as expected. I see the Content-Length = 0 and the filter doesn't buffer any data.

Expectation

I was expecting to see a non-zero content length and x-envoy-auth-partial-body header in the logs.

Repro Steps

Run the Dev version of Envoy which have this feature and make the following cURL request:

curl -X POST -H 'Accept: */*' -H 'Accept-Encoding: gzip, deflate, br' -H 'Connection: keep-alive' -H 'Content-Length: 44' -H 'Content-Type: application/x-www-form-urlencoded' -H 'User-Agent: python-requests/2.31.0' -d 'grant_type=client_credentials&scope=all-apis' http://localhost:10000/oidc/v1/token

Config

admin:
  address:
    socket_address:
      protocol: TCP
      address: 127.0.0.1
      port_value: 9901
static_resources:
  listeners:
  - name: main_listener
    address:
      socket_address:
        protocol: TCP
        address: 0.0.0.0
        port_value: 10000
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: ingress_http
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              domains: ["*"]
              routes:
              - match:
                  prefix: "/"
                route:
                  cluster: upstream
                typed_per_filter_config:
                  envoy.filters.http.ext_authz:
                    "@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthzPerRoute
                    check_settings:
                      with_request_body:
                        max_request_bytes: 4096
                        allow_partial_message: true
          http_filters:
          - name: envoy.filters.http.ext_authz
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz
              http_service:
                server_uri:
                  cluster: extauthz
                  timeout: 60s
                  uri: http://127.0.0.1:10001
              status_on_error:
                code: ServiceUnavailable
              transport_api_version: "v3"
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
  clusters:
  - name: extauthz
    connect_timeout: 5.0s
    type: strict_dns
    lb_policy: round_robin
    protocol_selection: USE_DOWNSTREAM_PROTOCOL
    load_assignment:
      cluster_name: extauthz
      endpoints:
        lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: 127.0.0.1
                port_value: 10001
  - name: upstream
    connect_timeout: 5.0s
    type: strict_dns
    lb_policy: round_robin
    protocol_selection: USE_DOWNSTREAM_PROTOCOL
    load_assignment:
      cluster_name: upstream
      endpoints:
        lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: 127.0.0.1
                port_value: 10002

Logs

[2023-12-19 02:44:30.325][28][debug][conn_handler] [source/extensions/listener_managers/listener_manager/active_tcp_listener.cc:159] [Tags: "ConnectionId":"0"] new connection from 192.168.65.1:47915
[2023-12-19 02:44:30.334][28][debug][http] [source/common/http/conn_manager_impl.cc:391] [Tags: "ConnectionId":"0"] new stream
[2023-12-19 02:44:30.346][28][debug][http] [source/common/http/conn_manager_impl.cc:1194] [Tags: "ConnectionId":"0","StreamId":"4457926979748916328"] request headers complete (end_stream=false):
':authority', 'localhost:10000'
':path', '/oidc/v1/token'
':method', 'POST'
'accept', '*/*'
'accept-encoding', 'gzip, deflate, br'
'connection', 'keep-alive'
'content-length', '44'
'content-type', 'application/x-www-form-urlencoded'
'user-agent', 'python-requests/2.31.0'

[2023-12-19 02:44:30.351][28][debug][connection] [./source/common/network/connection_impl.h:98] [Tags: "ConnectionId":"0"] current connecting state: false
[2023-12-19 02:44:30.354][28][debug][ext_authz] [source/extensions/filters/http/ext_authz/ext_authz.cc:131] [Tags: "ConnectionId":"0","StreamId":"4457926979748916328"] ext_authz filter is buffering the request
[2023-12-19 02:44:30.356][28][debug][http] [source/common/http/conn_manager_impl.cc:1177] [Tags: "ConnectionId":"0","StreamId":"4457926979748916328"] request end stream
[2023-12-19 02:44:30.357][28][debug][ext_authz] [source/extensions/filters/http/ext_authz/ext_authz.cc:159] [Tags: "ConnectionId":"0","StreamId":"4457926979748916328"] ext_authz filter finished buffering the request since stream is ended
[2023-12-19 02:44:30.364][28][debug][router] [source/common/router/router.cc:528] [Tags: "ConnectionId":"0","StreamId":"8699100473878106352"] cluster 'extauthz' match for URL '/oidc/v1/token'
[2023-12-19 02:44:30.370][28][debug][router] [source/common/router/router.cc:740] [Tags: "ConnectionId":"0","StreamId":"8699100473878106352"] router decoding headers:
':authority', 'localhost:10000'
':method', 'POST'
':path', '/oidc/v1/token'
':scheme', 'http'
'content-length', '0'
'x-envoy-internal', 'true'
'x-envoy-expected-rq-timeout-ms', '60000'

[2023-12-19 02:44:30.371][28][debug][pool] [source/common/http/conn_pool_base.cc:78] queueing stream due to no available connections (ready=0 busy=0 connecting=0)
[2023-12-19 02:44:30.372][28][debug][pool] [source/common/conn_pool/conn_pool_base.cc:291] trying to create new connection
[2023-12-19 02:44:30.372][28][debug][pool] [source/common/conn_pool/conn_pool_base.cc:145] creating a new connection (connecting=0)
[2023-12-19 02:44:30.375][28][debug][connection] [./source/common/network/connection_impl.h:98] [Tags: "ConnectionId":"1"] current connecting state: true
[2023-12-19 02:44:30.375][28][debug][client] [source/common/http/codec_client.cc:57] [Tags: "ConnectionId":"1"] connecting
[2023-12-19 02:44:30.375][28][debug][connection] [source/common/network/connection_impl.cc:1009] [Tags: "ConnectionId":"1"] connecting to 127.0.0.1:10001
[2023-12-19 02:44:30.376][28][debug][connection] [source/common/network/connection_impl.cc:1028] [Tags: "ConnectionId":"1"] connection in progress
[2023-12-19 02:44:30.379][28][debug][connection] [source/common/network/connection_impl.cc:757] [Tags: "ConnectionId":"1"] delayed connect error: 111
[2023-12-19 02:44:30.379][28][debug][connection] [source/common/network/connection_impl.cc:278] [Tags: "ConnectionId":"1"] closing socket: 0
[2023-12-19 02:44:30.379][28][debug][client] [source/common/http/codec_client.cc:107] [Tags: "ConnectionId":"1"] disconnect. resetting 0 pending requests
[2023-12-19 02:44:30.380][28][debug][pool] [source/common/conn_pool/conn_pool_base.cc:484] [Tags: "ConnectionId":"1"] client disconnected, failure reason: delayed connect error: 111
[2023-12-19 02:44:30.381][28][debug][router] [source/common/router/router.cc:1331] [Tags: "ConnectionId":"0","StreamId":"8699100473878106352"] upstream reset: reset reason: remote connection failure, transport failure reason: delayed connect error: 111
[2023-12-19 02:44:30.385][28][debug][http] [source/common/http/async_client_impl.cc:106] async http request response headers (end_stream=false):
':status', '503'
'content-length', '152'
'content-type', 'text/plain'

[agrawroh]

Closing as the fix has been merged. Thanks, @ggreenway for taking a look!