Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Large working ingress working with Modsecurity set to DetectionOnly, but ignored when set On #10115

Closed
markhley opened this issue Jun 20, 2023 · 31 comments
Closed

Large working ingress working with Modsecurity set to DetectionOnly, but ignored when set On #10115

markhley opened this issue Jun 20, 2023 · 31 comments
Assignees
@strongjz
Labels
kind/bug Categorizes issue or PR as related to a bug. priority/backlog Higher priority than priority/awaiting-more-evidence. triage/accepted Indicates an issue or PR is ready to be actively worked on. triage/needs-information Indicates an issue needs more information in order to work on it.

Comments

@markhley
Copy link
Contributor

markhley commented Jun 20, 2023 •
edited
Loading

What happened:

We have several Azure AKS clusters installed using Nginx as the ingress controller and using the built-in Modsecurity for WAF functionallity. All clusters have Modsecurity set to "SecRuleEngine On" and all are fully functional except our largest production environment. The major difference is the large ingress definition in the production cluster. It has many host definitions, each having a certificate secret assigned. The large ingress definition works fine with Modsecurity set to "SecRuleEngine DetectionOnly", but once we set Modsecuriy to "Modsecurity set to "SecRuleEngine On", the ingress definition is ignored and all sites are down. While Modsecurity is set "SecRuleEngine On" in this production environment, If I cut the number of hosts down to just a few, it works again. When original large ingress is in place, I can only get sites up with Modsecutiy set to "SecRuleEngine DetectionOnly"

What you expected to happen:

I expect large ingress to work with all sites responding when Modsecurity is set "SecRuleEngine On"

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):

Release:       v1.6.4
 Build:         69e8833858fb6bda12a44990f1d5eaa7b13f4b75
 Repository:    https://github.com/kubernetes/ingress-nginx
 nginx version: nginx/1.21.6

Kubernetes version (use kubectl version):

v1.25.6

Environment:

  • Cloud provider or hardware configuration: Azure
  • OS (e.g. from /etc/os-release):
  • Kernel (e.g. uname -a):
  • Install tools:
    • Please mention how/where was the cluster created like kubeadm/kops/minikube/kind etc. AKS cluster
  • Basic cluster related info:
    • kubectl version v1.25.6
    • kubectl get nodes -o wide
NAME                                STATUS   ROLES   AGE     VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
aks-consent-40674510-vmss000008     Ready    agent   50d     v1.25.6   10.0.114.159   <none>        Ubuntu 22.04.2 LTS   5.15.0-1035-azure   containerd://1.6.18+azure-1
aks-consent-40674510-vmss000009     Ready    agent   50d     v1.25.6   10.0.114.190   <none>        Ubuntu 22.04.2 LTS   5.15.0-1035-azure   containerd://1.6.18+azure-1
aks-pool1-40674510-vmss0001km       Ready    agent   50d     v1.25.6   10.0.114.148   <none>        Ubuntu 22.04.2 LTS   5.15.0-1035-azure   containerd://1.6.18+azure-1
aks-pool1-40674510-vmss0001ry       Ready    agent   50d     v1.25.6   10.0.112.115   <none>        Ubuntu 22.04.2 LTS   5.15.0-1035-azure   containerd://1.6.18+azure-1
aks-pool1-40674510-vmss0002e0       Ready    agent   27d     v1.25.6   10.0.113.77    <none>        Ubuntu 22.04.2 LTS   5.15.0-1035-azure   containerd://1.6.18+azure-1
aks-pool1-40674510-vmss0002vx       Ready    agent   7d19h   v1.25.6   10.0.113.82    <none>        Ubuntu 22.04.2 LTS   5.15.0>1035-azure   containerd://1.6.18+azure-1
aks-pool1-40674510-vmss0003ao       Ready    agent   59m     v1.25.6   10.0.112.5     <none>        Ubuntu 22.04.2 LTS   5.15.0-1035-azure   containerd://1.6.18+azure-1
aks-pool1-40674510-vmss0003ap       Ready    agent   58m     v1.25.6   10.0.113.193   <none>        Ubuntu 22.04.2 LTS   5.15.0-1035-azure   containerd://1.6.18+azure-1
aks-pool1-40674510-vmss0003aq       Ready    agent   58m     v1.25.6   10.0.115.146   <none>        Ubuntu 22.04.2 LTS   5.15.0-1035-azure   containerd://1.6.18+azure-1
aks-truyodata-28269258-vmss000000   Ready    agent   50d     v1.25.6   10.0.115.38    <none>        Ubuntu 22.04.2 LTS   5.15.0-1035-azure   containerd://1.6.18+azure-1
aks-truyodata-28269258-vmss000001   Ready    agent   50d     v1.25.6   10.0.115.67    <none>        Ubuntu 22.04.2 LTS   5.15.0-1035-azure   containerd://1.6.18+azure-1
aks-truyodata-28269258-vmss000002   Ready    agent   50d     v1.25.6   10.0.115.96    <none>        Ubuntu 22.04.2 LTS   5.15.0-1035-azure   containerd://1.6.18+azure-1
  • How was the ingress-nginx-controller installed:
    • If helm was used then please show output of helm ls -A | grep -i ingress
      ngress-nginx-4.5.2 1.6.4
    • If helm was used then please show output of helm -n <ingresscontrollernamepspace> get values <helmreleasename>
    USER-SUPPLIED VALUES:
controller:
  config:
    enable-modsecurity: true
    enable-owasp-modsecurity-crs: true
    http-snippet: |
      map $geoip2_country_code $allowed_country {
       default yes;
       CN no;
       SG no;
       PH no;
      }
    location-snippet: |
      # location block
      if ($allowed_country = no) {
       return 403;
      }
    proxy-body-size: 100m
    use-geoip: false
    use-geoip2: true
  extraArgs:
    default-ssl-certificate: default/truyo-com-certificate
    maxmind-edition-ids: GeoLite2-City,GeoLite2-ASN,GeoLite2-Country
  extraVolumeMounts:
  - mountPath: /etc/nginx/modsecurity/modsecurity.conf
    name: modsecurity-config
    subPath: modsecurity.conf
  - mountPath: /etc/nginx/owasp-modsecurity-crs/crs-setup.conf
    name: crs-setup-config
    subPath: crs-setup.conf
  - mountPath: /etc/nginx/owasp-modsecurity-crs/rules/RESPONSE-999-EXCLUSION-RULES-AFTER-CRS.conf
    name: response-999-exclusion-rules-after-crs-config
    subPath: RESPONSE-999-EXCLUSION-RULES-AFTER-CRS.conf
  extraVolumes:
  - configMap:
      name: modsecurity-config
    name: modsecurity-config
  - configMap:
      name: crs-setup-config
    name: crs-setup-config
  - configMap:
      name: response-999-exclusion-rules-after-crs-config
    name: response-999-exclusion-rules-after-crs-config
  kind: DaemonSet
  maxmindLicenseKey: **********************
  service:
    externalTrafficPolicy: Local

  • If helm was not used, then copy/paste the complete precise command used to install the controller, along with the flags and options used

  • if you have more than one instance of the ingress-nginx-controller installed in the same cluster, please provide details for all the instances

  • Current State of the controller:

    • kubectl describe ingressclasses
    Name:         nginx
    Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=nginx
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=ingress-nginx
              app.kubernetes.io/part-of=ingress-nginx
              app.kubernetes.io/version=1.6.4
              helm.sh/chart=ingress-nginx-4.5.2
              helm.toolkit.fluxcd.io/name=nginx
              helm.toolkit.fluxcd.io/namespace=nginx
    Annotations:  meta.helm.sh/release-name: nginx
              meta.helm.sh/release-namespace: nginx
   Controller:   k8s.io/ingress-nginx
   Events:       <none>
  • kubectl -n <ingresscontrollernamespace> get all -A -o wide
NAME                                       READY   STATUS    RESTARTS   AGE   IP             NODE                            NOMINATED NODE   READINESS GATES
pod/nginx-ingress-nginx-controller-45txv   1/1     Running   0          18d   10.0.114.250   aks-pool1-40674510-vmss0001km   <none>           <none>
pod/nginx-ingress-nginx-controller-5jtdf   1/1     Running   0          31m   10.0.112.37    aks-pool1-40674510-vmss0003bx   <none>           <none>
pod/nginx-ingress-nginx-controller-697cd   1/1     Running   0          18d   10.0.112.216   aks-pool1-40674510-vmss0001ry   <none>           <none>
pod/nginx-ingress-nginx-controller-78ksx   1/1     Running   0          18d   10.0.114.60    aks-pool1-40674510-vmss0002e0   <none>           <none>
pod/nginx-ingress-nginx-controller-fvnwg   1/1     Running   0          31m   10.0.112.237   aks-pool1-40674510-vmss0003by   <none>           <none>
pod/nginx-ingress-nginx-controller-mj569   1/1     Running   0          8d    10.0.113.126   aks-pool1-40674510-vmss0002vx   <none>           <none>
pod/nginx-ingress-nginx-controller-qzhq2   1/1     Running   0          31m   10.0.116.153   aks-pool1-40674510-vmss0003c0   <none>           <none>
pod/nginx-ingress-nginx-controller-vl979   1/1     Running   0          31m   10.0.114.28    aks-pool1-40674510-vmss0003bz   <none>           <none>

NAME                                               TYPE           CLUSTER-IP       EXTERNAL-IP     PORT(S)                      AGE    SELECTOR
service/nginx-ingress-nginx-controller             LoadBalancer   10.100.193.182   *.*.*.*   80:30497/TCP,443:32132/TCP   585d   app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginx,app.kubernetes.io/name=ingress-nginx
service/nginx-ingress-nginx-controller-admission   ClusterIP      10.100.212.203   <none>          443/TCP                      585d   app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginx,app.kubernetes.io/name=ingress-nginx

NAME                                            DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE    CONTAINERS   IMAGES                                                                                                                    SELECTOR
daemonset.apps/nginx-ingress-nginx-controller   8         8         8       8            8           kubernetes.io/os=linux   534d   controller   registry.k8s.io/ingress-nginx/controller:v1.6.4@sha256:15be4666c53052484dd2992efacf2f50ea77a78ae8aa21ccd91af6baaa7ea22f   app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginx,app.kubernetes.io/name=ingress-nginx
  • kubectl -n <ingresscontrollernamespace> describe po <ingresscontrollerpodname>
Name:         nginx-ingress-nginx-controller-45txv
Namespace:    nginx
Priority:     0
Node:         aks-pool1-40674510-vmss0001km/10.0.114.148
Start Time:   Sun, 28 May 2023 22:38:07 -0700
Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=nginx
              app.kubernetes.io/name=ingress-nginx
              controller-revision-hash=7fd584fcdb
              pod-template-generation=10
Annotations:  kubectl.kubernetes.io/restartedAt: 2023-05-28T22:34:37-07:00
Status:       Running
IP:           10.0.114.250
IPs:
  IP:           10.0.114.250
Controlled By:  DaemonSet/nginx-ingress-nginx-controller
Containers:
  controller:
    Container ID:  containerd://c0b151ca145f365f36e7c22454ddba076877cd31e27ac9a7d784391e9a207639
    Image:         registry.k8s.io/ingress-nginx/controller:v1.6.4@sha256:15be4666c53052484dd2992efacf2f50ea77a78ae8aa21ccd91af6baaa7ea22f
    Image ID:      registry.k8s.io/ingress-nginx/controller@sha256:15be4666c53052484dd2992efacf2f50ea77a78ae8aa21ccd91af6baaa7ea22f
    Ports:         80/TCP, 443/TCP, 8443/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP
    Args:
      /nginx-ingress-controller
      --publish-service=$(POD_NAMESPACE)/nginx-ingress-nginx-controller
      --election-id=nginx-ingress-nginx-leader
      --controller-class=k8s.io/ingress-nginx
      --ingress-class=nginx
      --configmap=$(POD_NAMESPACE)/nginx-ingress-nginx-controller
      --validating-webhook=:8443
      --validating-webhook-certificate=/usr/local/certificates/cert
      --validating-webhook-key=/usr/local/certificates/key
      --maxmind-license-key=********************
      --default-ssl-certificate=default/truyo-com-certificate
      --maxmind-edition-ids=GeoLite2-City,GeoLite2-ASN,GeoLite2-Country
    State:          Running
      Started:      Sun, 28 May 2023 22:38:08 -0700
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:      100m
      memory:   90Mi
    Liveness:   http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
    Readiness:  http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:       nginx-ingress-nginx-controller-45txv (v1:metadata.name)
      POD_NAMESPACE:  nginx (v1:metadata.namespace)
      LD_PRELOAD:     /usr/local/lib/libmimalloc.so
    Mounts:
      /etc/nginx/modsecurity/modsecurity.conf from modsecurity-config (rw,path="modsecurity.conf")
      /etc/nginx/owasp-modsecurity-crs/crs-setup.conf from crs-setup-config (rw,path="crs-setup.conf")
      /etc/nginx/owasp-modsecurity-crs/rules/RESPONSE-999-EXCLUSION-RULES-AFTER-CRS.conf from response-999-exclusion-rules-after-crs-config (rw,path="RESPONSE-999-EXCLUSION-RULES-AFTER-CRS.conf")
      /usr/local/certificates/ from webhook-cert (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-2586k (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  webhook-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  nginx-ingress-nginx-admission
    Optional:    false
  modsecurity-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      modsecurity-config
    Optional:  false
  crs-setup-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      crs-setup-config
    Optional:  false
  response-999-exclusion-rules-after-crs-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      response-999-exclusion-rules-after-crs-config
    Optional:  false
  kube-api-access-2586k:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists
                             node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                             node.kubernetes.io/unreachable:NoExecute op=Exists
                             node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:                      <none>
  • kubectl -n <ingresscontrollernamespace> describe svc <ingresscontrollerservicename>
Name:                     nginx-ingress-nginx-controller
Namespace:                nginx
Labels:                   app.kubernetes.io/component=controller
                          app.kubernetes.io/instance=nginx
                          app.kubernetes.io/managed-by=Helm
                          app.kubernetes.io/name=ingress-nginx
                          app.kubernetes.io/part-of=ingress-nginx
                          app.kubernetes.io/version=1.6.4
                          helm.sh/chart=ingress-nginx-4.5.2
                          helm.toolkit.fluxcd.io/name=nginx
                          helm.toolkit.fluxcd.io/namespace=nginx
Annotations:              meta.helm.sh/release-name: nginx
                          meta.helm.sh/release-namespace: nginx
Selector:                 app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginx,app.kubernetes.io/name=ingress-nginx
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.100.193.182
IPs:                      10.100.193.182
LoadBalancer Ingress:     *.*.*.*
Port:                     http  80/TCP
TargetPort:               http/TCP
NodePort:                 http  30497/TCP
Endpoints:                10.0.112.216:80,10.0.112.237:80,10.0.112.37:80 + 5 more...
Port:                     https  443/TCP
TargetPort:               https/TCP
NodePort:                 https  32132/TCP
Endpoints:                10.0.112.216:443,10.0.112.237:443,10.0.112.37:443 + 5 more...
Session Affinity:         None
External Traffic Policy:  Local
HealthCheck NodePort:     30639
Events:
  Type     Reason                        Age                   From                       Message
  ----     ------                        ----                  ----                       -------
  Warning  FailedToUpdateEndpointSlices  54m                   endpoint-slice-controller  Error updating Endpoint Slices for Service nginx/nginx-ingress-nginx-controller: [skipping Pod nginx-ingress-nginx-controller-zgh4c for Service nginx/nginx-ingress-nginx-controller: Node aks-pool1-40674510-vmss0003bp Not Found, skipping Pod nginx-ingress-nginx-controller-tm4c9 for Service nginx/nginx-ingress-nginx-controller: Node aks-pool1-40674510-vmss0003bt Not Found]
  Warning  FailedToUpdateEndpointSlices  53m (x4 over 54m)     endpoint-slice-controller  Error updating Endpoint Slices for Service nginx/nginx-ingress-nginx-controller: skipping Pod nginx-ingress-nginx-controller-zgh4c for Service nginx/nginx-ingress-nginx-controller: Node aks-pool1-40674510-vmss0003bp Not Found
  Normal   UpdatedLoadBalancer           17m (x29 over 4h29m)  service-controller         Updated load balancer with new hosts
  Warning  FailedToUpdateEndpointSlices  6m44s                 endpoint-slice-controller  Error updating Endpoint Slices for Service nginx/nginx-ingress-nginx-controller: skipping Pod nginx-ingress-nginx-controller-65b46 for Service nginx/nginx-ingress-nginx-controller: Node aks-pool1-40674510-vmss0003bv Not Found
  • Current state of ingress object, if applicable:
    • kubectl -n <appnnamespace> get all,ing -o wide
    • kubectl -n <appnamespace> describe ing <ingressname>
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    kubernetes.io/ingress.class: "nginx"
    ingress.kubernetes.io/force-ssl-redirect: "true"
  name: gdpr-gateway
  labels:
    app: kong
spec:
  tls:
  - hosts:
      - "*.wildcard.com"
    secretName: wildcard-com-certificate
  - hosts:
      - "customer1.com"
    secretName: customer1-certificate
  - hosts:
      - "customer2.com"
    secretName: customer2-tls-certs-secret
  - hosts:
      - "customer3.com"
    secretName: customer3-tls-certs-secret
  - hosts:
     - "customer4.com"
    secretName: customer4-tls-certs-secret
  - hosts:
     - "customer5.com" 
    secretName: customer5-certificate
  - hosts:
     - "customer6.com" 
    secretName: customer6-tls-certs-secret
  - hosts:
     - "customer7.com" 
    secretName: customer7-tls-certs-secret
  - hosts:
     - "customer8.com" 
    secretName: customer8-certificate
  - hosts:
     - "customer9.com" 
    secretName: customer9-certificate
  - hosts:
     - "customer10.com"
    secretName: customer10-certificate
  - hosts:
     - "customer11.com"
    secretName: customer11-tls-certs-secret
  - hosts:
     - "customer12.com"
    secretName: customer12-certificate
  - hosts:
     - "customer13.com"
    secretName: customer13-certificate
  - hosts:
     - "customer14.com"
    secretName: wildcard-com-certificate
  - hosts:
     - "customer15.com"
    secretName: customer15-certificate
  - hosts:
     - "customer16.com"
    secretName: customer16-certificate
  - hosts:
     - "customer17.com"
    secretName: customer17-certificate
  - hosts:
     - "customer18.com"
    secretName: customer18-certificate
  - hosts:
     - "customer19.com"
    secretName: customer19-certificate
  - hosts:
     - "customer20.com"
    secretName: customer20-tls-certs-secret
  - hosts:
     - "customer21.com"
    secretName: customer21-tls-certs-secret
  - hosts:
     - "customer22.com"
    secretName: customer22-certificate
  - hosts:
     - "customer23.com"
    secretName: customer23-tls-certs-secret
  - hosts:
     - "customer24.com"
    secretName: customer24-tls-certs-secret
  - hosts:
     - "customer25.com"
    secretName: customer25-tls-certs-secret
  - hosts:
     - "customer26.com"
    secretName: customer26-certificate
  - hosts:
     - "customer27.com"
    secretName: customer27-certificate
  - hosts:
     - "customer28.carvana.com"
    secretName: customer28-certificate
  - hosts:
     - "customer29.com"
    secretName: customer29-certificate 
  - hosts:
     - "customer30.com"
    secretName: customer30-certificate    
  rules:
    - host: "*.wildcard.com"
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: gdpr-gateway-kong-proxy
              port:
                number: 80
    - host: "customer1.com"
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: gdpr-gateway-kong-proxy
              port:
                number: 80
    - host: "customer2.com"
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: gdpr-gateway-kong-proxy
              port:
                number: 80
    - host: "customer3.com"
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: gdpr-gateway-kong-proxy
              port:
                number: 80
    - host: "customer4.com"
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: gdpr-gateway-kong-proxy
              port:
                number: 80
    - host: "customer5.com"
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: gdpr-gateway-kong-proxy
              port:
                number: 80
    - host: "customer6.com"
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: gdpr-gateway-kong-proxy
              port:
                number: 80
    - host: "customer7.com"
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: gdpr-gateway-kong-proxy
              port:
                number: 80
    - host: "customer8.com"
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: gdpr-gateway-kong-proxy
              port:
                number: 80
    - host: "customer9.com"
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: gdpr-gateway-kong-proxy
              port:
                number: 80
    - host: "customer10.com"                                                                                                                                              
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: gdpr-gateway-kong-proxy
              port:
                number: 80
    - host: "customer11.com"                                                                                                                                              
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: gdpr-gateway-kong-proxy
              port:
                number: 80
    - host: "customer12.com"                                                                                                                                              
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: gdpr-gateway-kong-proxy
              port:
                number: 80
    - host: "customer13.com"                                                                                                                                              
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: gdpr-gateway-kong-proxy
              port:
                number: 80
    - host: "customer14.com"                                                                                                                                              
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: gdpr-gateway-kong-proxy
              port:
                number: 80
    - host: "customer15.com"                                                                                                                                              
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: gdpr-gateway-kong-proxy
              port:
                number: 80
    - host: "customer16.com"
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: gdpr-gateway-kong-proxy
              port:
                number: 80
    - host: "customer17.com"
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: gdpr-gateway-kong-proxy
              port:
                number: 80                         
    - host: "customer18.com"
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: gdpr-gateway-kong-proxy
              port:
                number: 80               
    - host: "customer19.com"
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: gdpr-gateway-kong-proxy
              port:
                number: 80
    - host: "customer20.com"
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: gdpr-gateway-kong-proxy
              port:
                number: 80
    - host: "customer21.com"
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: gdpr-gateway-kong-proxy
              port:
                number: 80
    - host: "customer22.com"
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: gdpr-gateway-kong-proxy
              port:
                number: 80
    - host: "customer23.com"
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: gdpr-gateway-kong-proxy
              port:
                number: 80
    - host: "customer24.com"
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: gdpr-gateway-kong-proxy
              port:
                number: 80
    - host: "customer25.com"
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: gdpr-gateway-kong-proxy
              port:
                number: 80
    - host: "customer26.com"
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: gdpr-gateway-kong-proxy
              port:
                number: 80
    - host: "customer27.com"
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: gdpr-gateway-kong-proxy
              port:
                number: 80
    - host: "customer28.com"
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: gdpr-gateway-kong-proxy
              port:
                number: 80
    - host: "customer29.com"
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: gdpr-gateway-kong-proxy
              port:
                number: 80                
    - host: "customer30.com"
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: gdpr-gateway-kong-proxy
              port:
                number: 80

  • If applicable, then, your complete and exact curl/grpcurl command (redacted if required) and the reponse to the curl/grpcurl command with the -v flag

  • Others:

    • Any other related information like ;
      • copy/paste of the snippet (if applicable)
      • kubectl describe ... of any custom configmap(s) created and in use

See attached configmap yaml files

- Any other related information that may help

How to reproduce this issue:

Anything else we need to know:
@markhley markhley added the kind/bug Categorizes issue or PR as related to a bug. label Jun 20, 2023
@k8s-ci-robot k8s-ci-robot added needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-priority labels Jun 20, 2023
@markhley
Copy link
Contributor Author

01-modsecurity-config.yaml.txt
01-RESPONSE-999-EXCLUSION-RULES-AFTER-CRS-config.yaml.txt
01-crs-setup-config.yaml.txt

@longwuyuan
Copy link
Contributor

@markhley the relevant info visible is the daemonset choice with 8 odd replicas. The info not available is the log messages of the controller pods that relate to modsecurity use-case and reload.

Increased resource usage and contention is expected obviously based on the very nature of WAF and 8 pods with live traffic. But specifics can be known after you post logs of the controller pods. Please redact as info you may wan to hide is visible here. And post the logs to a gist and provide the link here.

/remove-kind bug

@k8s-ci-robot k8s-ci-robot added needs-kind Indicates a PR lacks a `kind/foo` label and requires one. and removed kind/bug Categorizes issue or PR as related to a bug. labels Jun 21, 2023
@longwuyuan
Copy link
Contributor

Additionally, splitting those hostname + path combo of rules into individual ingress objects will reduce the size of dataset involved in reconciliations. If possible please do change to small dedicated ingress objects.

@markhley
Copy link
Contributor Author

I will try splitting ingress objects into individual ingress definitions. I will have to schedule prod change/test in order to do so. I will let you know results

@longwuyuan
Copy link
Contributor

ok. hopefully it will reduce resource usage in reconcile and maybe mitigate the problem.

@longwuyuan
Copy link
Contributor

/triage needs-information

@k8s-ci-robot k8s-ci-robot added the triage/needs-information Indicates an issue needs more information in order to work on it. label Jun 22, 2023
@strongjz
Copy link
Member

/assign @strongjz

@markhley
Copy link
Contributor Author

As far logs, our sites are hit at large volume and logs are hard to parse. I may have to turn off access logs to see if we can get better diagnostics for this situation on the error. I will wait to see results this weekend when we split the ingresses into individual files. If that fails, I will try to get logs bundled

@markhley
Copy link
Contributor Author

Over weekend, we reconfigured our large ingress into individual ingress definitions. This change was successful with Modsecurity set to "SecRuleEngine DetectionOnly". All URLs responded successfully. We then tried Modsecurity set to "SecRuleEngine On" and immediately started having same issues with all our ingresses. The primary relevant error in log is as follows "[lua] certificate.lua:244: call(): certificate not found, falling back to fake certificate for hostname: customer1.com, context: ssl_certificate_by_lua*" I am attaching error log file for inspection

@markhley
Copy link
Contributor Author

nginx-errors.log

@markhley
Copy link
Contributor Author

/assign @strongjz

@strongjz - Have you had a chance to look at error logs yet?

@strongjz
Copy link
Member

strongjz commented Jul 6, 2023

We discussed this on the community call and believe we found the issue.

Mod security is working properly but it is blocking the controllers internal request to update the nginx.conf

POST /configuration/servers HTTP/1.1", host: "127.0.0.1:10246 is the port and uri for the controller to update the conf when there are changes. So enabling modsecurity is blocking that request for being too large. We are going to put in a fix in 1.8.2 to disable modsec for that server block in the nginx.tmpl template.

2023/06/26 05:24:50 [error] 1912#1912: *32231 [client 127.0.0.1] ModSecurity: Access denied with code 400 (phase 2). Matched "Operator `Eq' with parameter `0' against variable `REQBODY_ERROR' (Value: `1' ) [file "/etc/nginx/modsecurity/modsecurity.conf"] [line "76"] [id "200002"] [rev ""] [msg "Failed to parse request body."] [data "Request body excluding files is bigger than the maximum expected."] [severity "2"] [ver ""] [maturity "0"] [accuracy "0"] [hostname "127.0.0.1"] [uri "/configuration/servers"] [unique_id "168775709078.157738"] [ref "v0,1"], client: 127.0.0.1, server: , request: "POST /configuration/servers HTTP/1.1", host: "127.0.0.1:10246"
W0626 05:24:50.571596       7 controller.go:236] Dynamic reconfiguration failed (retrying; 6 retries left): unexpected error code: 400

@strongjz
Copy link
Member

strongjz commented Jul 6, 2023

Here is the server block in the template https://github.com/kubernetes/ingress-nginx/blob/main/rootfs/etc/nginx/template/nginx.tmpl#L707

if you set the modsecurity off; in that block it should fix till we have the fix in place.

@strongjz
Copy link
Member

strongjz commented Jul 6, 2023

/triage accepted
/kind bug
/priority backlog

@k8s-ci-robot k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. kind/bug Categorizes issue or PR as related to a bug. priority/backlog Higher priority than priority/awaiting-more-evidence. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-kind Indicates a PR lacks a `kind/foo` label and requires one. needs-priority labels Jul 6, 2023
@markhley
Copy link
Contributor Author

markhley commented Jul 6, 2023 via email

@strongjz strongjz moved this to In Progress in [SIG Network] Ingress NGINX Jul 6, 2023
@markhley
Copy link
Contributor Author

markhley commented Jul 6, 2023

Sunday, we are going to try your hotfix recommendation by replacing nginx.tmpl "server" section with custom configmap and setting "modsecurity off;" per your recommendation. I will update the ticket with results

@rikatz
Copy link
Contributor

rikatz commented Jul 6, 2023

@markhley if this fixes, and you are willing to contribute, can you please open a PR on nginx.tmpl to add the check as well?

Thanks!

@markhley
Copy link
Contributor Author

markhley commented Jul 7, 2023

@rikatz - Not sure what check you are referring to, but here is adjustment we are testing Sunday in nginx.tmpl in "default server" section around line 707.

I will do PR if you guys want if we have success

# default server, used for NGINX healthcheck and access to nginx stats
server {
    modsecurity off;
    
    listen 127.0.0.1:{{ .StatusPort }};
    set $proxy_upstream_name "internal";

    keepalive_timeout 0;
    gzip off;

    access_log off;

@markhley
Copy link
Contributor Author

Success - with Modsecurity turned off in nginx.tmpl "server" section, we were able to set Modsecurity to "SecRuleEngine On" and all ingresses responded appropriately. I tested a mock WAF violation against site and received expected 403 from Modsecurity. Thanks for all the help on this. If you want me to submit this change as a PR, let me know and I will go through documented process.

@markhley
Copy link
Contributor Author

My assumption on check is you want something like this I found in another part of nginx.tmpl instead of hard-coded override

# Ensure that modsecurity will not run on an internal location as this is not accessible from outside
{{ if $all.Cfg.EnableModsecurity }}
modsecurity off;
{{ end }}

@rikatz
Copy link
Contributor

rikatz commented Jul 16, 2023

correct, I'm adding the fix here

@rikatz
Copy link
Contributor

rikatz commented Jul 16, 2023

@markhley actually do you want to make a PR and do a contribution? :) Otherwise I can deal with it, but the idea is what you did, if you want send the PR and I can tell you if we can get it merged.

Thanks for the contribution and the finding here!

@markhley
Copy link
Contributor Author

I will work on PR to get experience. My first open source PR

@hobti01
Copy link
Contributor

hobti01 commented Aug 8, 2023

We are also impacted by this. @markhley will you have the time to send a PR?

@markhley
Copy link
Contributor Author

markhley commented Aug 8, 2023 via email

@hobti01
Copy link
Contributor

hobti01 commented Aug 8, 2023

OK I understand. I don't want to discourage your participation and your analysis has been invaluable - but if you run into blockers I kindly ask for you to post here and we'll send a fix over.

@markhley
Copy link
Contributor Author

markhley commented Aug 9, 2023 via email

@hobti01
Copy link
Contributor

hobti01 commented Aug 10, 2023

clicking the contribute button I think is the easiest way
image

@markhley
Copy link
Contributor Author

markhley commented Aug 10, 2023 via email

@hobti01 hobti01 mentioned this issue Sep 5, 2023
Merged
10 tasks
@rikatz
Copy link
Contributor

rikatz commented Oct 15, 2023

/close

The PR above should have fixed it, feel free to reopen if this is still a problem

@k8s-ci-robot
Copy link
Contributor

@rikatz: Closing this issue.

In response to this:

/close

The PR above should have fixed it, feel free to reopen if this is still a problem

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@github-project-automation github-project-automation bot moved this from In Progress to Done in [SIG Network] Ingress NGINX Oct 15, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@strongjz strongjz

Labels
kind/bug Categorizes issue or PR as related to a bug. priority/backlog Higher priority than priority/awaiting-more-evidence. triage/accepted Indicates an issue or PR is ready to be actively worked on. triage/needs-information Indicates an issue needs more information in order to work on it.
Projects
[SIG Network] Ingress NGINX
Archived in project
Milestone
No milestone
Development

No branches or pull requests
6 participants
@strongjz @hobti01 @longwuyuan @rikatz @markhley @k8s-ci-robot