Can't deploy Jaeger on AWS EKS using Calico

[klubi]

Describe the bug
Jaeger deployment on AWS EKS running Calico CNI fails with below error:

Internal error occurred: failed calling webhook "mjaeger.kb.io": Post "https://jaeger-operator-webhook-service.monitoring.svc:443/mutate-jaegertracing-io-v1-jaeger?timeout=10s": Address is not allowed

Expected behavior
Jaeger instance deploys successfully.

Additional context
I faced similar issue installing jaeger-operator. Fix was on cert-manager side, I had to set hostNetwork: true on deployment.

[frzifus]

hi @klubi, could you share the output of kubectl describe MutatingWebhookConfiguration mjaeger.kb.io?

update:

thats might be the issue you are running into: https://medium.com/@denisstortisilva/kubernetes-eks-calico-and-custom-admission-webhooks-a2956b49bd0d

[klubi]

I used:

kubectl describe mutatingwebhookconfigurations.admissionregistration.k8s.io jaeger-operator-mutating-webhook-configuration

got

Name:         jaeger-operator-mutating-webhook-configuration
Namespace:
Labels:       app.kubernetes.io/managed-by=spinnaker
              app.kubernetes.io/name=jaeger-operator
              name=jaeger-operator
Annotations:  artifact.spinnaker.io/location:
              artifact.spinnaker.io/name: jaeger-operator-mutating-webhook-configuration
              artifact.spinnaker.io/type: kubernetes/mutatingWebhookConfiguration
              artifact.spinnaker.io/version:
              cert-manager.io/inject-ca-from: monitoring/jaeger-operator-serving-cert
              moniker.spinnaker.io/application: jaeger
              moniker.spinnaker.io/cluster: mutatingWebhookConfiguration jaeger-operator-mutating-webhook-configuration
API Version:  admissionregistration.k8s.io/v1
Kind:         MutatingWebhookConfiguration
Metadata:
  Creation Timestamp:  2022-06-27T08:11:40Z
  Generation:          2
  Managed Fields:
    API Version:  admissionregistration.k8s.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:webhooks:
        k:{"name":"deployment.sidecar-injector.jaegertracing.io"}:
          f:clientConfig:
            f:caBundle:
        k:{"name":"mjaeger.kb.io"}:
          f:clientConfig:
            f:caBundle:
    Manager:      cainjector
    Operation:    Update
    Time:         2022-06-27T08:49:43Z
    API Version:  admissionregistration.k8s.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:artifact.spinnaker.io/location:
          f:artifact.spinnaker.io/name:
          f:artifact.spinnaker.io/type:
          f:artifact.spinnaker.io/version:
          f:cert-manager.io/inject-ca-from:
          f:kubectl.kubernetes.io/last-applied-configuration:
          f:moniker.spinnaker.io/application:
          f:moniker.spinnaker.io/cluster:
        f:labels:
          .:
          f:app.kubernetes.io/managed-by:
          f:app.kubernetes.io/name:
          f:name:
      f:webhooks:
        .:
        k:{"name":"deployment.sidecar-injector.jaegertracing.io"}:
          .:
          f:admissionReviewVersions:
          f:clientConfig:
            .:
            f:service:
              .:
              f:name:
              f:namespace:
              f:path:
              f:port:
          f:failurePolicy:
          f:matchPolicy:
          f:name:
          f:namespaceSelector:
          f:objectSelector:
          f:reinvocationPolicy:
          f:rules:
          f:sideEffects:
          f:timeoutSeconds:
        k:{"name":"mjaeger.kb.io"}:
          .:
          f:admissionReviewVersions:
          f:clientConfig:
            .:
            f:service:
              .:
              f:name:
              f:namespace:
              f:path:
              f:port:
          f:failurePolicy:
          f:matchPolicy:
          f:name:
          f:namespaceSelector:
          f:objectSelector:
          f:reinvocationPolicy:
          f:rules:
          f:sideEffects:
          f:timeoutSeconds:
    Manager:         kubectl
    Operation:       Update
    Time:            2022-06-27T09:11:48Z
  Resource Version:  496147993
  UID:               11988c97-ebd3-4640-9624-024121f11a7d
Webhooks:
  Admission Review Versions:
    v1
  Client Config:
    Ca Bundle:  <REDACTED>
    Service:
      Name:        jaeger-operator-webhook-service
      Namespace:   monitoring
      Path:        /mutate-v1-deployment
      Port:        443
  Failure Policy:  Ignore
  Match Policy:    Equivalent
  Name:            deployment.sidecar-injector.jaegertracing.io
  Namespace Selector:
  Object Selector:
    Match Expressions:
      Key:       app.kubernetes.io/name
      Operator:  NotIn
      Values:
        jaeger-operator
  Reinvocation Policy:  Never
  Rules:
    API Groups:
      apps
    API Versions:
      v1
    Operations:
      CREATE
      UPDATE
    Resources:
      deployments
    Scope:          *
  Side Effects:     None
  Timeout Seconds:  10
  Admission Review Versions:
    v1
  Client Config:
    Ca Bundle:  <REDACTED>
    Service:
      Name:        jaeger-operator-webhook-service
      Namespace:   monitoring
      Path:        /mutate-jaegertracing-io-v1-jaeger
      Port:        443
  Failure Policy:  Fail
  Match Policy:    Equivalent
  Name:            mjaeger.kb.io
  Namespace Selector:
  Object Selector:
  Reinvocation Policy:  Never
  Rules:
    API Groups:
      jaegertracing.io
    API Versions:
      v1
    Operations:
      CREATE
      UPDATE
    Resources:
      jaegers
    Scope:          *
  Side Effects:     None
  Timeout Seconds:  10
Events:             <none>

[klubi]

And yes, link you've sent @frzifus seems to describe what I see.

Question is... where to set hostNetwork, and if it's possible to change port used by webhook in case I run into conflicts.
I did run into conflict with cert-manager, but switching ports on that one was very easy.

Edit:
I believe I know how to set it... 9443 port seems to be safe in my case, and metrics port (8081) looks to be easy to change...

[klubi]

Ok... I was able to make it work.
Had to:

• add hostNetwork: true to operator deployment spec
• change health port from 8081 to something else (I used 10270)
• change kube-rbac-proxy secure port from 8443 to something else (I used 10280)

I had to leave webhook listener port on 9443 because I wasn't able to find a flag that would allow operator to listen on different port than 9443. I was lucky enough, I don't have anything else listening on that port...

[frzifus]

Awesome, thanks for sharing. Would you like to extend our docs and/or introduce a flag for changing the webhook listener port?

[klubi]

Yeah, I can do that.