ModSec "Phase 4" interventions not working

[patrick-laa]

Service name

Submit a crime form / Assess a crime form

Service environment

• Dev / Development
• Staging
• Prod / Production
• Other

Impact on the service

If ModSec correctly identifies a problem with the response body of any HTTP request, and attempts to prevent that body from being returned to an end user, its intervention is unsuccessful, and the problematic body does make it to the end user. This means we are not receiving full protection from security threats.

Problem description

• Our systems use ModSec, configured in a fairly standard way.
• In "Phase 4" ModSec scans the response body of an HTTP request that has been routed from the ingress to a deployment pod
• If ModSec determines that there is an issue with that body, it will attempt to flag up that issue, overriding the original status code with a 403 and removing the response body so that it is not returned to the end user.
• However, when ModSec triggers in phase 4 and attempts to do this, what we see is "header already sent while sending to client" in the ingress logs, the response status recorded by the ingress is 500 and, crucially, the response body is still sent back to the client
• For example, this file is (wrongly) considered by ModSec to contain Java source code, so ModSec intervenes when you try to request it: https://crm457-1721-b-expl-nscc-provider-dev.cloud-platform.service.justice.gov.uk/assets/file-upload-b450e66e.js.map. However, if you request it via, e.g. the browser, you will notice that you still receive it in full. If you then check the Kibana ingress logs in the namespace laa-submit-crime-forms-dev you will see the corresponding request has an upstream status of 200 and a status of 500.
• This is a known issue of ModSecurity-nginx
• I have tried to ascertain if anyone using Cloud Platform is able to get ModSec to successfully intervene in phase 4 and have drawn a blank.

Contact person

@Patrick Gleeson (on Slack) / [email protected]. If Patrick's contract has ended by the time this gets addressed, also try @Jo Gilder, @Joel Sugarman or @Bex Appleyard.

[jaskaransarkaria]

@patrick-laa @jo Gilder @joel Sugarman @bex Appleyard let me know if you want to step through the reproducible test I have set up.

Findings:

I have reproduced the issue in a test cluster and have got closer to mimicking the full functionality. The closet I can get is:

• If a user uses http1.1 for the request AND sends a request that triggers an invalid response body THEN the response body is missing because there is an "Empty reply from server" and the connection is closed, see below

* Ignoring the response-body
* Leftovers after chunking: 11 bytes
* Connection #0 to host ssi.cp-0312-0954.cloud-platform.service.justice.gov.uk left intact
* Clear auth, redirects to port from 80 to 443
* Issue another request to this URL: 'https://ssi.cp-0312-0954.cloud-platform.service.justice.gov.uk/'
* Host ssi.cp-0312-0954.cloud-platform.service.justice.gov.uk:443 was resolved.
* IPv6: (none)
* IPv4: xxxxxxx
*   Trying 13.41.136.31:443...
* Connected to ssi.cp-0312-0954.cloud-platform.service.justice.gov.uk (xxxxx) port 443
* ALPN: curl offers http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384 / [blank] / UNDEF
* ALPN: server accepted http/1.1
* Server certificate:
*  subject: CN=*.apps.cp-0312-0954.cloud-platform.service.justice.gov.uk
*  start date: Dec  3 09:27:34 2024 GMT
*  expire date: Mar  3 09:27:33 2025 GMT
*  subjectAltName: host "ssi.cp-0312-0954.cloud-platform.service.justice.gov.uk" matched cert's "*.cp-0312-0954.cloud-platform.service.justice.gov.uk"
*  issuer: C=US; O=Let's Encrypt; CN=R10
*  SSL certificate verify ok.
* using HTTP/1.x
> GET / HTTP/1.1
> Host: ssi.cp-0312-0954.cloud-platform.service.justice.gov.uk
> User-Agent: jaazz
> Accept: */*
>
* Empty reply from server
* Closing connection
curl: (52) Empty reply from server

The above isn't ideal (we would want to receive the status 403 too) and we still see header already sent while sending response to client

here's a template ingress modsec snippet:

    nginx.ingress.kubernetes.io/enable-modsecurity: "true"
    nginx.ingress.kubernetes.io/proxy-buffering: "on"
    nginx.ingress.kubernetes.io/modsecurity-snippet: |
      SecRuleEngine On
      SecDefaultAction "phase:2,pass,log,tag:github_team=blah"
      SecResponseBodyAccess On
      SecResponseBodyMimeType text/plain text/html text/xml
      SecRule RESPONSE_BODY "@rx <p>Visitor User agent: jaazz</p>" "id:1000137,phase:4,deny,nolog,status:403"

I think this is the most relevant issue that reflects our problem:

owasp-modsecurity/ModSecurity-nginx#254

Looks, like response body filtering isn't yet fully implemented (still todos in the code for the response body tests) and the nginx-connector hasn't addressed only sending the response headers once the response body has been analysed.

Next Steps:

Because response body filtering isn't implemented I will turn off SecResponseBodyAccess and prevent it from being logged, until this is fixed upstream. This will have a positive impact on performance.

Other relevant issues:
owasp-modsecurity/ModSecurity-nginx#14
owasp-modsecurity/ModSecurity-nginx#41
owasp-modsecurity/ModSecurity-nginx#61
owasp-modsecurity/ModSecurity-nginx#63
owasp-modsecurity/ModSecurity-nginx#84
owasp-modsecurity/ModSecurity-nginx#93
owasp-modsecurity/ModSecurity-nginx#97