500s in gateway logs after enabling sidecars in authorino & limitador, and STRICT PeerAuthentication

[david-martin]

After doing a make local-setup locally, then following https://docs.kuadrant.io/dev/kuadrant-operator/doc/user-guides/secure-protect-connect/, I got to the point of a working AuthPolicy & RatelimitPolicy.

I then enabled mtls (via sidecar injection) in authorino and limitador in an attempt to have encrypted traffic from envoy to/from the kuadrant components.
Here's the cmds I ran.

kubectl -n kuadrant-system patch deployment authorino \
  -p '{"spec":{"template":{"metadata":{"labels":{"sidecar.istio.io/inject":"true"}}}}}'


kubectl -n kuadrant-system patch deployment limitador-limitador \
  -p '{"spec":{"template":{"metadata":{"labels":{"sidecar.istio.io/inject":"true"}}}}}'

kubectl apply -f - <<EOF
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: kuadrant-system
spec:
  mtls:
    mode: STRICT
EOF

However, after enabling this, all responses are now 500s.
I bumped up the log level to debug in the gateway, using this cmd:

kubectl exec $(kubectl -n kuadrant-system get po -l istio.io/gateway-name=api-gateway -o name
pod/api-gateway-istio-649dc76fc-nklds) -n kuadrant-system -- curl -X POST http://localhost:15000/logging?level=debug

Here's a snippet of logs
https://gist.github.com/david-martin/90104685603ecab49aad45cf595d2c92

Here's the Pod spec & config_dump of the gateway
https://gist.github.com/david-martin/43fdbbede84df3b156609081a9b6387f

Discussion on slack at https://kubernetes.slack.com/archives/C05J0D0V525/p1732621388340329

At this time it's unclear if the issue is with the envoy filter configuration that kuadrant manages, or something else.

[david-martin]

I was able to get it working locally with some patching of the envoyfilters, though this isn't a viable solution as the filters are maintained by the kuadrant-operator.
I added this to both thekuadrant-auth-service & kuadrant-ratelimit-service envoyfilters:

          transport_socket:
            name: envoy.transport_sockets.tls
            typed_config:
              '@type': type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
              common_tls_context:
                tls_certificate_sds_secret_configs:
                - name: default
                  sds_config:
                    api_config_source:
                      api_type: GRPC
                      grpc_services:
                      - envoy_grpc:
                          cluster_name: sds-grpc

From chat on slack, this may not be validating certificates as well, so something may be missing.

From @guicassolato

For mTLS, I think you need both, the tls_certificates* stanza and the validation_context* one

https://istio.io/latest/docs/ops/configuration/traffic-management/tls-configuration/#outbound

Also, quote from @eguzki

One of the issues with this is that for adding mtls client config, you need to use gateway provider APIs to do that (either Istio or EnvoyGateway). Kuadrant does not own (manage) the gateways. And today this is still under developement. EnvoyGateway has a slightly more flexible way of doing that. But we need to explore the way to provide certificates for the mtls use cases.

Our use case is not common. This is not about setting up backend connections. This is about setting up some comm's between wasm module and some service

[david-martin]

Closing issue with follow up manual guide in #1054 and RFC for automated way to enable mTLS in Kuadrant/architecture#110