TLS passthrough issues after upgrading from 0.15.3 to 1.0.1

[brenix]

What steps did you take and what happened:

In the process of upgrading contour/envoy from v0.15.3/v1.11.2 to v1.0.1/v1.12.2, we are seeing TLS related issues for a TCP proxy with TLS passthrough enabled (defined via IngressRoute or HTTPProxy)

The proxy appears to be working, but the TLS negotiation between a GRPC client and server service are failing with messages:

Client:

transport: authentication handshake failed: tls: first record does not look like a TLS handshake

Server:

transport: http2Server.HandleStreams received bogus greeting from client

When reverting our deployment back to Contour 0.15.3, the service resumes connectivity again.

Below is an example of the HTTPProxy:

apiVersion: projectcontour.io/v1
kind: HTTPProxy
metadata:
  name: server-service
spec:
  virtualhost:
    fqdn: server.example.com
    tls:
      passthrough: true
  tcpproxy:
    services:
    - name: server-service
      port: 443

What did you expect to happen:

Without any changes being made to the IngressRoute/HTTPProxy, I'd expect the services to connect without TLS errors as they did prior to the upgrade.

Anything else you would like to add:

FWIW, the connection is being routed through a AWS NLB:

Client service --> NLB --> Envoy/Contour --> Server service

As far as I can see, the only related changes since 0.15.3 were made in #1450 - Thinking something there may have changed the behavior and possibly the issue we are seeing.

Environment:

• Contour version: 1.0.1
• Envoy version: 1.12.2
• Kubernetes version: 1.16.4
• Kubernetes installer & version: kubeadm
• Cloud provider or hardware configuration: AWS

[brenix]

Update:

We upgraded to 1.1.0 and did not see any change, the issue remains.

However, after adding a route with protocol: h2c to the spec, the service was able to establish connectivity.

Example:

spec:
  routes:
  - match: /
    services:
    - name: server-service
      port: 443
      protocol: h2c

This is a GRPC service - We continue to see TLS errors if the protocol was set to tls or h2, but it appears to work with h2c

[brenix]

After looking into this further, our service had an older annotation to define the upstream protocol. This was rather updated to reflect the new projectcontour domain and correct protocol. Closing this for now

[stevesloka]

Hey @brenix, so switching the service annotation from contour.heptio.com to projectcontour.io fixed your problem?

[brenix]

Hey @brenix, so switching the service annotation from contour.heptio.com to projectcontour.io fixed your problem?

Yes, as well as switching the protocol to h2c.

We went from:

contour.heptio.com/upstream-protocol.h2: "443"

to

projectcontour.io/upstream-protocol.h2c: "443"

which fixed it.