connect native to a sidecar proxy fails with "Failed ALPN negotiation: Unable to find compatible protocol"

[rgf2004]

The sidecar proxy returns null/empty applicationProtocol during the TLS negotiation.

• I'm trying to establish a native integration with a gRPC java service running in nomad and has a sidecar proxy (envoy proxy).
• The issue happens during the TLS negotiation that the proxy replies with null/empty applicationProtocol.
• I tried to update the envoy configuration DownstreamTlsContext.CommonTlsContext.alpn_protocols for the public listener to add h2 as the application protocol using this consul configuration parameter however this didn't work for the following reasons:

Reason - 1: My configuration is not reflected in the envoy configuration. I assume this because consul overwrites it by the Connect TLS certificates as per this documentation

Every FilterChain added to the listener will have its TlsContext overridden by the Connect TLS certificates 
and validation context. This means there is no way to override Connect's mutual TLS for the public listener.

Reason - 2: Even if I override the public listener configuration, which I don't know how yet:), I can't find a way how to get the dynamic port value that proxy gets in listener.address.socket_address.port_value when it is created without overriding.

Java stack trace

io.grpc.StatusRuntimeException: UNAVAILABLE: Connection closed while performing protocol negotiation for [SslHandler#0, ProtocolNegotiators$ClientTlsHandler#0, WriteBufferingAndExceptionHandler#0, DefaultChannelPipeline$TailContext#0]
	at io.grpc.Status.asRuntimeException(Status.java:526)
	at io.grpc.netty.WriteBufferingAndExceptionHandler.channelInactive(WriteBufferingAndExceptionHandler.java:81)
	...
17:04:24.182 [grpc-nio-worker-ELG-1-4] DEBUG io.netty.handler.ssl.SslHandler.setHandshakeSuccess(1829) - [id: 0x46522a81, L:/127.0.0.1:57819 - R:/127.0.0.1:28430] HANDSHAKEN: protocol:TLSv1.3 cipher suite:TLS_AES_128_GCM_SHA256
17:04:24.182 [grpc-nio-worker-ELG-1-4] DEBUG io.grpc.netty.ProtocolNegotiators.logSslEngineDetails(902) - TLS negotiation failed.
SSLEngine Details: [
    OpenSSL, Version: 0x1010107f (BoringSSL), ALPN supported: true
    TLS Protocol: TLSv1.3
    Application Protocol: null <--
    Need Client Auth: false
    Want Client Auth: false
    Supported protocols=[SSLv2Hello, TLSv1, TLSv1.1, TLSv1.2, TLSv1.3]
    Enabled protocols=[SSLv2Hello, TLSv1.3]
    Supported ciphers=[ECDHE-ECDSA-AES128-GCM-SHA256, ECDHE-RSA-AES128-GCM-SHA256, ECDHE-ECDSA-AES256-GCM-SHA384, ECDHE-RSA-AES256-GCM-SHA384, ECDHE-ECDSA-CHACHA20-POLY1305, ECDHE-RSA-CHACHA20-POLY1305, ECDHE-PSK-CHACHA20-POLY1305, ECDHE-ECDSA-AES128-SHA, ECDHE-RSA-AES128-SHA, ECDHE-PSK-AES128-CBC-SHA, ECDHE-ECDSA-AES256-SHA, ECDHE-RSA-AES256-SHA, ECDHE-PSK-AES256-CBC-SHA, AES128-GCM-SHA256, AES256-GCM-SHA384, AES128-SHA, PSK-AES128-CBC-SHA, AES256-SHA, PSK-AES256-CBC-SHA, DES-CBC3-SHA, TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256, AEAD-AES128-GCM-SHA256, AEAD-AES256-GCM-SHA384, AEAD-CHACHA20-POLY1305-SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, SSL_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, SSL_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, SSL_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, SSL_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, SSL_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256, SSL_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_PSK_WITH_CHACHA20_POLY1305_SHA256, SSL_ECDHE_PSK_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, SSL_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, SSL_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_PSK_WITH_AES_128_CBC_SHA, SSL_ECDHE_PSK_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, SSL_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, SSL_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_PSK_WITH_AES_256_CBC_SHA, SSL_ECDHE_PSK_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA256, SSL_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_GCM_SHA384, SSL_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_128_CBC_SHA, SSL_RSA_WITH_AES_128_CBC_SHA, TLS_PSK_WITH_AES_128_CBC_SHA, SSL_PSK_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, SSL_RSA_WITH_AES_256_CBC_SHA, TLS_PSK_WITH_AES_256_CBC_SHA, SSL_PSK_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, null]
    Enabled ciphers=[TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256]
]
io.grpc.StatusRuntimeException: UNAVAILABLE: Failed ALPN negotiation: Unable to find compatible protocol
	at io.grpc.Status.asRuntimeException(Status.java:526)
	...
17:04:24.183 [grpc-nio-worker-ELG-1-4] DEBUG io.grpc.ChannelLogger.logOnly(131) - [Subchannel<3>: (127.0.0.1:28430)] NettyClientTransport<5>: (/127.0.0.1:28430) SHUTDOWN with UNAVAILABLE(Failed ALPN negotiation: Unable to find compatible protocol
Channel Pipeline: [SslHandler#0, ProtocolNegotiators$ClientTlsHandler#0, WriteBufferingAndExceptionHandler#0, DefaultChannelPipeline$TailContext#0])

Consul version v1.11.1
Envoy version 74c221751138e5add71e0738d40092434b76a7cf/1.13.4/Modified/RELEASE/BoringSSL
Nomad version Nomad v1.2.3 (a79efc8422082c4790046c3f5ad92c542592a54f)

Please advise how I can configure the sidecar proxy (the envoy proxy) via consul to advertise the application protocol h2 to let the TLS negotiation succeed.

[Amier3]

Hey @rgf2004

Keeping you updated on this, we have a PR #12170 that should resolve this issue.