Route with partial tls_certificate breaks Envoy and logs private key

[stevesloka]

Title: Route with partial tls_certificate breaks Envoy and logs private key

Description:
We had an issue come up with Contour where a user was using Let's Encrypt with CertManager to provision TLS certificates. During the provisioning process, they hit a rate-limit in Let's Encrypt. This resulted in a private key being provisioned but that's all.

When Contour saw this Kubernetes secret get created it pushed that down to Envoy. When this happened Envoy rejects the configuration and emits this error log:

[2019-04-26 16:32:29.677][000001][warning][config] [bazel-out/k8-opt/bin/source/common/config/_virtual_includes/grpc_mux_subscription_lib/common/config/grpc_mux_subscription_impl.h:70] gRPC config for type.googleapis.com/envoy.api.v2.Listener rejected: Proto constraint validation failed (ListenerValidationError.FilterChains[i]: ["embedded message failed validation"] | caused by FilterChainValidationError.TlsContext: ["embedded message failed validation"] | caused by DownstreamTlsContextValidationError.CommonTlsContext: ["embedded message failed validation"] | caused by CommonTlsContextValidationError.TlsCertificates[i]: ["embedded message failed validation"] | caused by TlsCertificateValidationError.CertificateChain: ["embedded message failed validation"] | caused by field: "specifier", reason: is required): name: "ingress_https"

When this happens, Envoy dumps the entire config to the log file including the private keys.

The biggest issue is that once we're in this state, Envoy's configuration is frozen in time and does not follow new Ingress/Endpoint updates. If the Envoy restarts for any reason, it will fail to come back online at all.

The Contour team is going to ask CertManager if this is a good scenario to be put into as well as we're going to handle this case specifically in Contour so we can mitigate, but I thought it would be good to raise here and see if I could help Envoy not get to this state or if I was missing a configuration item that would prevent this from happening.

Relevant Links:
Contour issue: projectcontour/contour#1051

[mattklein123]

@stevesloka I think this is roughly a duplicate of #4757

[alyssawilk]

Yeah, looks like a dup to me as well. @stevesloka please reopen if we are mistaken. If not, we would welcome a fix to this - #4757 is marked as help

[stevesloka]

Thanks @mattklein123 & @alyssawilk! I'd love to help out, let me go ping the other issue.