EKS 1.11 eks.2 + Istio 1.1.1 + Cilium 1.4.2, Automatic sidecar injection failed: Address is not allowed, failed calling admission webhook "sidecar-injector.istio.io"

[brant4test]

Steps to reproduce the issue

1). Install an AWS EKS cluster (1.11)
Kubernetes Version 1.11
Platform Version eks.2

2). Install cilium-etcd-operator on AWS ESK

kubectl -n kube-system set env ds aws-node AWS_VPC_K8S_CNI_EXTERNALSNAT=true
kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/v1.4/examples/kubernetes/1.11/cilium.yaml

3). Install istio-1.1.1 on AWS ESK

curl -L https://git.io/getLatestIstio | ISTIO_VERSION=1.1.1 sh -
cd istio-1.1.1
export PATH=$PWD/bin:$PATH
kubectl apply -f install/kubernetes/helm/helm-service-account.yaml
helm init --service-account tiller
helm install install/kubernetes/helm/istio-init --name istio-init --namespace istio-system
kubectl get crds | grep 'istio.io\|certmanager.k8s.io' | wc -l
53
helm upgrade istio install/kubernetes/helm/istio --namespace istio-system --set global.mtls.enabled=false -i --set global.proxy.includeIPRanges="10.0.0.0/8\,172.20.0.0/12"

4). Install sample Nginx for testing automatic sidecar injection

$ kubectl create ns nginx
$ kubectl label namespace nginx istio-injection=enabled
$ kubectl create deployment --image nginx nginx -n nginx
$ kubectl expose deployment nginx --port=80 --type=LoadBalancer -n nginx

Then you would see the issue as follows.

$ kubectl get deploy  -n nginx
NAME    DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
nginx   1         0         0            0           15m
$ kubectl get deploy  -n nginx -oyaml
...
  status:
    conditions:
    - lastTransitionTime: "2019-04-01T09:30:05Z"
      lastUpdateTime: "2019-04-01T09:30:05Z"
      message: Deployment does not have minimum availability.
      reason: MinimumReplicasUnavailable
      status: "False"
      type: Available
    - lastTransitionTime: "2019-04-01T09:30:05Z"
      lastUpdateTime: "2019-04-01T09:30:05Z"
      message: 'Internal error occurred: failed calling admission webhook "sidecar-injector.istio.io":
        Post https://istio-sidecar-injector.istio-system.svc:443/inject?timeout=30s:
        Address is not allowed'
      reason: FailedCreate
      status: "True"
      type: ReplicaFailure
    - lastTransitionTime: "2019-04-01T09:40:06Z"
      lastUpdateTime: "2019-04-01T09:40:06Z"
      message: ReplicaSet "nginx-78f5d695bd" has timed out progressing.
      reason: ProgressDeadlineExceeded
      status: "False"
      type: Progressing
    observedGeneration: 1
    unavailableReplicas: 1

Debug 1
Disabled Cilium policy explicitly, didn't help.

$ kubectl exec -it -n kube-system cilium-94bm9 -- bash

root@ip-172-30-23-2:~# cilium config
Conntrack                Enabled
ConntrackAccounting      Enabled
ConntrackLocal           Disabled
Debug                    Disabled
DebugLB                  Disabled
DropNotification         Enabled
MonitorAggregationLevel  None
PolicyTracing            Disabled
TraceNotification        Enabled
k8s-configuration        
k8s-endpoint             
PolicyEnforcement        default

root@ip-172-30-23-2:~# exit

$ kubectl get netpol --all-namespaces
No resources found.

$ kubectl get cnp --all-namespaces
No resources found.

$ kubectl edit ds cilium -n kube-system
--enable-policy=never

$ kubectl exec -it -n kube-system cilium-7thlm -- cilium config
Conntrack                Enabled
ConntrackAccounting      Enabled
ConntrackLocal           Disabled
Debug                    Disabled
DebugLB                  Disabled
DropNotification         Enabled
MonitorAggregationLevel  None
PolicyTracing            Disabled
TraceNotification        Enabled
k8s-configuration        
k8s-endpoint             
PolicyEnforcement        never

Re-tried sample Nginx for testing automatic sidecar injection, Got same error as before.

Debug 2
Disabled configValidation, Manual sidecar injection succeeded with sample Nginx test.

helm upgrade istio install/kubernetes/helm/istio --namespace istio-system --set global.mtls.enabled=false -i --set global.proxy.includeIPRanges="10.0.0.0/8\,172.20.0.0/20" --set global.configValidation=false

So which part of code blocked admission webhook in "EKS+Cilium+Istio"? Any suggestions? Thanks!

P.S. a related question on stackoverflow.

[amstee]

Hey, I ran in the same issue with Calico as a CNI on EKS, this is surely related to this.
After installing istio I get this error:

Internal error occurred: failed calling admission webhook \"mixer.validation.istio.io\": Post https://istio-galley.istio-system.svc:443/admitmixer?timeout=30s: Address is not allowed

My theory is :
This is due to the fact that the Calico CNI is present only on my worker nodes (Pods CIDR is 192.168.../16) and the control plane still run the AWS CNI as I don't have control over this with EKS.

Meaning that the webhook (running from the control plane) isn't allowed to communicate with my service istio-galley.istio-system.svc having an IP outside of the VPC.

[anemptyair]

We also got the same issue durning install metrics-server with Cilium 1.4.2 on EKS 1.11 eks.2.
We have expose 443 between eks control and worker node. Installed metrics-server via the way suggested by EKS.

Get error that seem cause by connection issue of 443 port.

kubectl get apiservice v1beta1.metrics.k8s.io -o yaml

Message: no response from https://10.99.121.153:443: Get https://10.99.121.153:443: dial tcp 10.99.121.153:443: Address is not allowed
Reason: FailedDiscoveryCheck
Status: False