when proxying gRPC, Kong does not set the HTTP2 :authority header

[mflendrich]

Summary

When proxying gRPC requests, Kong sets the Host header of an outgoing request but not the :authority header.

To me, this is incorrect behavior, because:

• per RFC 7540 section 8.1.2.3:

  Clients that generate HTTP/2 requests directly SHOULD use the ":authority" pseudo-header field instead of the Host header field.

  and gRPC requests can be considered "directly generated HTTP/2 requests";

• at least some upstream gRPC applications that rely on the :authority header to perform host-based request routing fail to route such requests (for example Istio 1.7.0-rc.1 where routing is performed by Envoy)

Expected behavior: Kong setting :authority on gRPC requests to the upstream servers. Possibly this should extend to all HTTP/2 requests.

Steps To Reproduce

1. Run an nginx with debug logging as a dummy upstream gRPC server:

   sudo docker run --rm -it --name nginx -v $PWD/nginx.conf:/etc/nginx/nginx.conf:ro nginx:1.19.2 nginx-debug

   nginx.conf:

   worker_processes 1;
   daemon off;
   
   events {
   	use           epoll;
   	worker_connections  128;
   }
   
   http {
     log_format debug_format '$remote_addr - $remote_user [$time_local] '
   											'"$request" $status $body_bytes_sent '
   											'"$http_referer" "$http_user_agent" host "$host" hosthdr "$http_host"';
   	access_log /dev/stdout debug_format;
   	error_log /dev/stdout debug;
   	server {
   		listen        8080 http2;
   
   		location      / {
   			root      /data;
   		}
   	}
   }
   

2. Run kong 2.1.2 with the following config:

   kong.conf

   database=off
   daemon=off
   declarative_config=kong.yml
   
   proxy_listen=0.0.0.0:9080 http2

   kong.yml

   _format_version: "2.1"
   _transform: true
   services:
   - name: grpc
     protocol: grpc
     host: 172.17.0.2 # this is the IP address of the nginx docker container
     port: 8080
     routes:
     - name: catch-all
       paths:
       - /
       protocols:
       - grpc

3. Make a gRPC request to Kong:

   grpcurl -v -plaintext -d '{}' 127.0.0.1:9080 proto.SomeService.ListSomething
   

4. Observe in nginx logs (grep for http2 header:) that there is no :authority header but there is host):

   got:

   2020/08/17 13:52:41 [debug] 28#28: *2 http2 header: "host: 172.17.0.2"
   

   expected but did not get:

   2020/08/17 13:52:41 [debug] 28#28: *2 http2 header: ":authority: 172.17.0.2"
   

Additional Details & Logs

• Kong version ($ kong version): 2.1.2
• Kong debug-level startup logs ($ kong start --vv) n/a
• Kong error logs (<KONG_PREFIX>/logs/error.log) n/a
• Kong configuration (the output of a GET request to Kong's Admin port - see
  https://docs.konghq.com/latest/admin-api/#retrieve-node-information) n/a
• Operating system: ubuntu 20.04

[bungle]

@gszr do you remember why we added:

grpc_set_header      Host               $upstream_host;

This will then not add :authority. I am not sure if this is a problem, but removing that could perhaps prevent some of the Kong features working. I am not sure can you set pseudo-headers with grpc_set_header, do you recall @gszr?

[gszr]

I do not recall, unfortunately. I started looking into this yesterday, but did not finish. By removing the Host header, I faced a different issue. Let's create a task to look deeper into this/

[bungle]

Sure. Just as a note Nginx ingress just removed the host:
kubernetes/ingress-nginx@d7b213d

[dndx]

Dig deeper into the above commit from @bungle reveals kubernetes/ingress-nginx#3706 (comment).

I think it was done intentionally when we developed the gRPC proxy feature as well. Maybe we can add a comment in the template file so to avoid further confusion on this.

[mflendrich]

What is the current status of this issue?

[gszr]

Hi @mflendrich,

@dndx and I have looked at it and due to how Nginx handles pseudo-headers, it is not currently possible to modify it from Kong using the APIs we have available. An Nginx patch will be required. We have a task planned to work on this. For the moment being, there are no workarounds.

[gszr]

Fixed by #6603