Mitigate http/2 attacks by authenticated clients #121197
Labels
lifecycle/frozen
Indicates that an issue or PR should not be auto-closed due to staleness.
sig/api-machinery
Categorizes an issue or PR as relevant to SIG API Machinery.
sig/network
Categorizes an issue or PR as relevant to SIG Network.
triage/accepted
Indicates an issue or PR is ready to be actively worked on.
Comments
k8s-ci-robot
added
needs-sig
|
/sig api-machinery |
|
@liggitt: The label(s) In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
k8s-ci-robot
added
sig/api-machinery
|
/sig network |
k8s-ci-robot
added
the
sig/network
liggitt
removed
the
triage/accepted
k8s-ci-robot
added
needs-triage
|
/milestone v1.29 |
|
Hello @enj , Bug triage team lead here. I want to check the status. |
simonpasquier
added a commit
to simonpasquier/prometheus-operator
that referenced
this issue
Oct 19, 2023
This change mitigates CVE-2023-44487 by disabling HTTP2 and forcing HTTP/1.1 until the Go standard library and golang.org/x/net are fully fixed. Right now, it is possible for authenticated and unauthenticated users to hold open HTTP2 connections and consume huge amounts of memory. Before this change: ``` curl -kv https://localhost:8443/metrics * Trying 127.0.0.1:8443... * Connected to localhost (127.0.0.1) port 8443 (#0) * ALPN: offers h2,http/1.1 [...] * ALPN: server accepted h2 [...] * using HTTP/2 * h2h3 [:method: GET] * h2h3 [:path: /metrics] * h2h3 [:scheme: https] * h2h3 [:authority: localhost:8443] * h2h3 [user-agent: curl/8.0.1] * h2h3 [accept: */*] * Using Stream ID: 1 (easy handle 0x5594d4614b10) [...] > GET /metrics HTTP/2 [...] ``` After this change: ``` curl -kv https://localhost:8443/metrics * Trying 127.0.0.1:8443... * Connected to localhost (127.0.0.1) port 8443 (#0) * ALPN: offers h2,http/1.1 [...] * ALPN: server accepted http/1.1 [...] * using HTTP/1.1 > GET /metrics HTTP/1.1 > Host: localhost:8443 > User-Agent: curl/8.0.1 > Accept: */* [...] < HTTP/1.1 200 OK [...] ``` See also: * kubernetes/kubernetes#121120 * kubernetes/kubernetes#121197 * golang/go#63417 (comment) Signed-off-by: Simon Pasquier <[email protected]>
simonpasquier
added a commit
to simonpasquier/prometheus-operator
that referenced
this issue
Oct 19, 2023
This change mitigates CVE-2023-44487 by disabling HTTP2 by default and forcing HTTP/1.1 until the Go standard library and golang.org/x/net are fully fixed. Right now, it is possible for authenticated and unauthenticated users to hold open HTTP2 connections and consume huge amounts of memory. It is possible to revert back the change by using the `--web.enable-http2` argument. Before this change: ``` curl -kv https://localhost:8443/metrics * Trying 127.0.0.1:8443... * Connected to localhost (127.0.0.1) port 8443 (#0) * ALPN: offers h2,http/1.1 [...] * ALPN: server accepted h2 [...] * using HTTP/2 * h2h3 [:method: GET] * h2h3 [:path: /metrics] * h2h3 [:scheme: https] * h2h3 [:authority: localhost:8443] * h2h3 [user-agent: curl/8.0.1] * h2h3 [accept: */*] * Using Stream ID: 1 (easy handle 0x5594d4614b10) [...] > GET /metrics HTTP/2 [...] ``` After this change: ``` curl -kv https://localhost:8443/metrics * Trying 127.0.0.1:8443... * Connected to localhost (127.0.0.1) port 8443 (#0) * ALPN: offers h2,http/1.1 [...] * ALPN: server accepted http/1.1 [...] * using HTTP/1.1 > GET /metrics HTTP/1.1 > Host: localhost:8443 > User-Agent: curl/8.0.1 > Accept: */* [...] < HTTP/1.1 200 OK [...] ``` See also: * kubernetes/kubernetes#121120 * kubernetes/kubernetes#121197 * golang/go#63417 (comment) Signed-off-by: Simon Pasquier <[email protected]>
5 tasks
simonpasquier
added a commit
to simonpasquier/prometheus-operator
that referenced
this issue
Oct 19, 2023
This change mitigates CVE-2023-44487 by disabling HTTP2 by default and forcing HTTP/1.1 until the Go standard library and golang.org/x/net are fully fixed. Right now, it is possible for authenticated and unauthenticated users to hold open HTTP2 connections and consume huge amounts of memory. It is possible to revert back the change by using the `--web.enable-http2` argument. Before this change: ``` curl -kv https://localhost:8443/metrics * Trying 127.0.0.1:8443... * Connected to localhost (127.0.0.1) port 8443 (#0) * ALPN: offers h2,http/1.1 [...] * ALPN: server accepted h2 [...] * using HTTP/2 * h2h3 [:method: GET] * h2h3 [:path: /metrics] * h2h3 [:scheme: https] * h2h3 [:authority: localhost:8443] * h2h3 [user-agent: curl/8.0.1] * h2h3 [accept: */*] * Using Stream ID: 1 (easy handle 0x5594d4614b10) [...] > GET /metrics HTTP/2 [...] ``` After this change: ``` curl -kv https://localhost:8443/metrics * Trying 127.0.0.1:8443... * Connected to localhost (127.0.0.1) port 8443 (#0) * ALPN: offers h2,http/1.1 [...] * ALPN: server accepted http/1.1 [...] * using HTTP/1.1 > GET /metrics HTTP/1.1 > Host: localhost:8443 > User-Agent: curl/8.0.1 > Accept: */* [...] < HTTP/1.1 200 OK [...] ``` See also: * kubernetes/kubernetes#121120 * kubernetes/kubernetes#121197 * golang/go#63417 (comment) Signed-off-by: Simon Pasquier <[email protected]>
simonpasquier
added a commit
to simonpasquier/cluster-monitoring-operator
that referenced
this issue
Oct 19, 2023
Commit b3b011b was not enough to prevent HTTP2 connections as demonstrated in kubernetes/kubernetes#121197. To prevent any risk with HTTP2, this change disables HTTP2 at the server level. There's no expected performance penalty because the web server is only used for scraping metrics. Signed-off-by: Simon Pasquier <[email protected]>
2 tasks
simonpasquier
added a commit
to simonpasquier/prometheus-operator
that referenced
this issue
Oct 19, 2023
This change mitigates CVE-2023-44487 by disabling HTTP2 by default and forcing HTTP/1.1 until the Go standard library and golang.org/x/net are fully fixed. Right now, it is possible for authenticated and unauthenticated users to hold open HTTP2 connections and consume huge amounts of memory. It is possible to revert back the change by using the `--web.enable-http2` argument. Before this change: ``` curl -kv https://localhost:8443/metrics * Trying 127.0.0.1:8443... * Connected to localhost (127.0.0.1) port 8443 (#0) * ALPN: offers h2,http/1.1 [...] * ALPN: server accepted h2 [...] * using HTTP/2 * h2h3 [:method: GET] * h2h3 [:path: /metrics] * h2h3 [:scheme: https] * h2h3 [:authority: localhost:8443] * h2h3 [user-agent: curl/8.0.1] * h2h3 [accept: */*] * Using Stream ID: 1 (easy handle 0x5594d4614b10) [...] > GET /metrics HTTP/2 [...] ``` After this change: ``` curl -kv https://localhost:8443/metrics * Trying 127.0.0.1:8443... * Connected to localhost (127.0.0.1) port 8443 (#0) * ALPN: offers h2,http/1.1 [...] * ALPN: server accepted http/1.1 [...] * using HTTP/1.1 > GET /metrics HTTP/1.1 > Host: localhost:8443 > User-Agent: curl/8.0.1 > Accept: */* [...] < HTTP/1.1 200 OK [...] ``` See also: * kubernetes/kubernetes#121120 * kubernetes/kubernetes#121197 * golang/go#63417 (comment) Signed-off-by: Simon Pasquier <[email protected]>
sadlerap
added a commit
to sadlerap/service-binding-runtime
that referenced
this issue
Oct 19, 2023
It appears that mitigating the recent http2 vulnerabilities (see CVE-2023-44487 and CVE-2023-39325) requires [more than just a library update to golang.org/x/net][1]. Until better mitigations have been developed, disable http2 in both the metrics and webhooks servers. [1]: kubernetes/kubernetes#121197 Signed-off-by: Andy Sadler <[email protected]>
1 task
|
@enj would it be possible to share how you tested this? (privately is also possible). We are working on mitigations for https://github.com/knative and I was testing with, but would be interested to see your approach. |
This was referenced Oct 26, 2023
Sorry, I cannot share the exploit code. Your approach is an okay approximation. You should however assume that a more sophisticated attack would lead to higher resource consumption. |
slashpai
pushed a commit
to slashpai/prometheus-operator
that referenced
this issue
Nov 3, 2023
This change mitigates CVE-2023-44487 by disabling HTTP2 by default and forcing HTTP/1.1 until the Go standard library and golang.org/x/net are fully fixed. Right now, it is possible for authenticated and unauthenticated users to hold open HTTP2 connections and consume huge amounts of memory. It is possible to revert back the change by using the `--web.enable-http2` argument. Before this change: ``` curl -kv https://localhost:8443/metrics * Trying 127.0.0.1:8443... * Connected to localhost (127.0.0.1) port 8443 (#0) * ALPN: offers h2,http/1.1 [...] * ALPN: server accepted h2 [...] * using HTTP/2 * h2h3 [:method: GET] * h2h3 [:path: /metrics] * h2h3 [:scheme: https] * h2h3 [:authority: localhost:8443] * h2h3 [user-agent: curl/8.0.1] * h2h3 [accept: */*] * Using Stream ID: 1 (easy handle 0x5594d4614b10) [...] > GET /metrics HTTP/2 [...] ``` After this change: ``` curl -kv https://localhost:8443/metrics * Trying 127.0.0.1:8443... * Connected to localhost (127.0.0.1) port 8443 (#0) * ALPN: offers h2,http/1.1 [...] * ALPN: server accepted http/1.1 [...] * using HTTP/1.1 > GET /metrics HTTP/1.1 > Host: localhost:8443 > User-Agent: curl/8.0.1 > Accept: */* [...] < HTTP/1.1 200 OK [...] ``` See also: * kubernetes/kubernetes#121120 * kubernetes/kubernetes#121197 * golang/go#63417 (comment) Signed-off-by: Simon Pasquier <[email protected]> (cherry picked from commit a62e814)
5 tasks
slashpai
pushed a commit
to slashpai/prometheus-operator
that referenced
this issue
Nov 3, 2023
This change mitigates CVE-2023-44487 by disabling HTTP2 by default and forcing HTTP/1.1 until the Go standard library and golang.org/x/net are fully fixed. Right now, it is possible for authenticated and unauthenticated users to hold open HTTP2 connections and consume huge amounts of memory. It is possible to revert back the change by using the `--web.enable-http2` argument. Before this change: ``` curl -kv https://localhost:8443/metrics * Trying 127.0.0.1:8443... * Connected to localhost (127.0.0.1) port 8443 (#0) * ALPN: offers h2,http/1.1 [...] * ALPN: server accepted h2 [...] * using HTTP/2 * h2h3 [:method: GET] * h2h3 [:path: /metrics] * h2h3 [:scheme: https] * h2h3 [:authority: localhost:8443] * h2h3 [user-agent: curl/8.0.1] * h2h3 [accept: */*] * Using Stream ID: 1 (easy handle 0x5594d4614b10) [...] > GET /metrics HTTP/2 [...] ``` After this change: ``` curl -kv https://localhost:8443/metrics * Trying 127.0.0.1:8443... * Connected to localhost (127.0.0.1) port 8443 (#0) * ALPN: offers h2,http/1.1 [...] * ALPN: server accepted http/1.1 [...] * using HTTP/1.1 > GET /metrics HTTP/1.1 > Host: localhost:8443 > User-Agent: curl/8.0.1 > Accept: */* [...] < HTTP/1.1 200 OK [...] ``` See also: * kubernetes/kubernetes#121120 * kubernetes/kubernetes#121197 * golang/go#63417 (comment) Signed-off-by: Simon Pasquier <[email protected]> (cherry picked from commit a62e814)
davidmogar
added a commit
to konflux-ci/release-service
that referenced
this issue
Nov 7, 2023
This is required to mitigate the CVE described here: kubernetes/kubernetes#121197 Signed-off-by: David Moreno García <[email protected]>
davidmogar
added a commit
to konflux-ci/release-service
that referenced
this issue
Nov 7, 2023
This is required to mitigate the CVE described here: kubernetes/kubernetes#121197 Signed-off-by: David Moreno García <[email protected]>
sadlerap
added a commit
to servicebinding/runtime
that referenced
this issue
Nov 7, 2023
* disable http2 for metrics and webhooks by default It appears that mitigating the recent http2 vulnerabilities (see CVE-2023-44487 and CVE-2023-39325) requires [more than just a library update to golang.org/x/net][1]. Until better mitigations have been developed, disable http2 in both the metrics and webhooks servers. [1]: kubernetes/kubernetes#121197 Signed-off-by: Andy Sadler <[email protected]> * cleanup http2 disabling methods Until better mitigations are in place, disable HTTP2 in all cases. Don't leave an option in place to re-enable it. Signed-off-by: Andy Sadler <[email protected]> * fix generated drift Signed-off-by: Andy Sadler <[email protected]> --------- Signed-off-by: Andy Sadler <[email protected]>
adinhodovic
pushed a commit
to adinhodovic/prometheus-operator
that referenced
this issue
Nov 13, 2023
This change mitigates CVE-2023-44487 by disabling HTTP2 by default and forcing HTTP/1.1 until the Go standard library and golang.org/x/net are fully fixed. Right now, it is possible for authenticated and unauthenticated users to hold open HTTP2 connections and consume huge amounts of memory. It is possible to revert back the change by using the `--web.enable-http2` argument. Before this change: ``` curl -kv https://localhost:8443/metrics * Trying 127.0.0.1:8443... * Connected to localhost (127.0.0.1) port 8443 (#0) * ALPN: offers h2,http/1.1 [...] * ALPN: server accepted h2 [...] * using HTTP/2 * h2h3 [:method: GET] * h2h3 [:path: /metrics] * h2h3 [:scheme: https] * h2h3 [:authority: localhost:8443] * h2h3 [user-agent: curl/8.0.1] * h2h3 [accept: */*] * Using Stream ID: 1 (easy handle 0x5594d4614b10) [...] > GET /metrics HTTP/2 [...] ``` After this change: ``` curl -kv https://localhost:8443/metrics * Trying 127.0.0.1:8443... * Connected to localhost (127.0.0.1) port 8443 (#0) * ALPN: offers h2,http/1.1 [...] * ALPN: server accepted http/1.1 [...] * using HTTP/1.1 > GET /metrics HTTP/1.1 > Host: localhost:8443 > User-Agent: curl/8.0.1 > Accept: */* [...] < HTTP/1.1 200 OK [...] ``` See also: * kubernetes/kubernetes#121120 * kubernetes/kubernetes#121197 * golang/go#63417 (comment) Signed-off-by: Simon Pasquier <[email protected]>
|
Hello! As this issue is tagged for 1.30, is it still planned for this release? |
|
Hello! Is it still planned for this release? |
|
We are past the code freeze stage. Moving this to the next release. /milestone v1.31 |
k8s-ci-robot
added
the
lifecycle/frozen
camilamacedo86
added a commit
to camilamacedo86/catalogd
that referenced
this issue
Dec 13, 2024
Ensure HTTP/2 is disabled by default for webhooks. Disabling HTTP/2 mitigates vulnerabilities associated with: - HTTP/2 Stream Cancellation (GHSA-qppj-fm5r-hxr3) - HTTP/2 Rapid Reset (GHSA-4374-p667-p6c8) While CVE fixes exist, they remain insufficient; disabling HTTP/2 helps reduce risks. For details, see: kubernetes/kubernetes#121197
camilamacedo86
added a commit
to camilamacedo86/catalogd
that referenced
this issue
Dec 13, 2024
Ensure HTTP/2 is disabled by default for webhooks. Disabling HTTP/2 mitigates vulnerabilities associated with: - HTTP/2 Stream Cancellation (GHSA-qppj-fm5r-hxr3) - HTTP/2 Rapid Reset (GHSA-4374-p667-p6c8) While CVE fixes exist, they remain insufficient; disabling HTTP/2 helps reduce risks. For details, see: kubernetes/kubernetes#121197
This was referenced Dec 17, 2024
Merged
github-merge-queue bot
pushed a commit
to operator-framework/catalogd
that referenced
this issue
Dec 17, 2024
Ensure HTTP/2 is disabled by default for webhooks. Disabling HTTP/2 mitigates vulnerabilities associated with: - HTTP/2 Stream Cancellation (GHSA-qppj-fm5r-hxr3) - HTTP/2 Rapid Reset (GHSA-4374-p667-p6c8) While CVE fixes exist, they remain insufficient; disabling HTTP/2 helps reduce risks. For details, see: kubernetes/kubernetes#121197
github-merge-queue bot
pushed a commit
to operator-framework/catalogd
that referenced
this issue
Dec 17, 2024
Ensure HTTP/2 is disabled by default for webhooks. Disabling HTTP/2 mitigates vulnerabilities associated with: - HTTP/2 Stream Cancellation (GHSA-qppj-fm5r-hxr3) - HTTP/2 Rapid Reset (GHSA-4374-p667-p6c8) While CVE fixes exist, they remain insufficient; disabling HTTP/2 helps reduce risks. For details, see: kubernetes/kubernetes#121197
camilamacedo86
added a commit
to camilamacedo86/catalogd
that referenced
this issue
Dec 18, 2024
Ensure HTTP/2 is disabled by default for webhooks. Disabling HTTP/2 mitigates vulnerabilities associated with: - HTTP/2 Stream Cancellation (GHSA-qppj-fm5r-hxr3) - HTTP/2 Rapid Reset (GHSA-4374-p667-p6c8) While CVE fixes exist, they remain insufficient; disabling HTTP/2 helps reduce risks. For details, see: kubernetes/kubernetes#121197
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
and others
golang/net@b225e7c does not sufficiently protect us from a malicious http/2 client. #121120 handles unauthenticated clients. This issue tracks mitigations for authenticated clients.
pprof svg from a single client with a single connection attempting to DOS a Kube API server that has max streams set to 100 (memory usage grew to 5 GB in a few mins before it stabilized - with multiple connections the API server would have easily OOM'd):
xref: golang/go#63417 (comment)
The text was updated successfully, but these errors were encountered: