Unexpected HTTP 431 response

[floone]

Describe the bug

Unexpected HTTP 431 response

When requesting resources via HTTPS, unexpected HTTP 431 (Request header fields too large) are returned. We observe this on production after migration to a managed openshift environment. The very same service worked as expected on the old environment.

We can reproduce it with curl. When the total request size is more than 7966 bytes (according to curl), a 431 ist returned instead of a 200.

According to our tests:

• it does not matter if bearer token is provided or not (our service uses authentication)
• it can be any resource, it does not have to exist
• it is the total request size that counts, not just size of the header lines

We see the error appeared only after the migration. We assume the new environment attaches more or different headers. On production we use a redhat build of Quarkus 2.13, however, it can be reproduced using 3.0.4.Final.

It seems to happen only when using HTTPS, and only with HTTP/2.

As a workaround, we disabled HTTP/2 using quarkus configuration:

quarkus.http.http2=false

Expected behavior

A HTTP 200 is returned

Actual behavior

A HTTP 431 is returned

How to Reproduce?

Use a plain quarkus app:

quarkus create app

Configure https:

cd /tmp
openssl req -newkey rsa:2048 -new -nodes -x509 -days 3650 -keyout key.pem -out cert.pem

application.properties:

quarkus.http.ssl.certificate.files=/tmp/cert.pem
quarkus.http.ssl.certificate.key-files=/tmp/key.pem

Request a resource via https.

If total bytes sent > 7966, a http 431 is returned:

$ curl -k 'https://localhost:8443/' \
      -H "x-foo: $(printf -- 'A%.0s' {1..7881})" \
      -o /dev/null -s -w '> bytes sent: %{size_request}' -vv 2>&1 |grep -E '(< HTTP|> bytes)'
< HTTP/2 200 
> bytes sent: 7966

$ curl -k 'https://localhost:8443/' \
      -H "x-foo: $(printf -- 'A%.0s' {1..7882})" \
      -o /dev/null -s -w '> bytes sent: %{size_request}' -vv 2>&1 |grep -E '(< HTTP|> bytes)'
< HTTP/2 431 
> bytes sent: 7967

However, it works (= HTTP 200) with https/http 1.1:

$ curl -k 'https://localhost:8443/' \
      -H "x-foo: $(printf -- 'A%.0s' {1..7882})" \
      -o /dev/null -s -w '> bytes sent: %{size_request}' -vv --http1.1 2>&1 |grep -E '(< HTTP|> bytes)'
< HTTP/1.1 200 OK
> bytes sent: 7969

It works (= HTTP 200) with plain http as well:

$ curl -k 'http://localhost:8080/' \
      -H "x-foo: $(printf -- 'A%.0s' {1..7882})" \
      -o /dev/null -s -w '> bytes sent: %{size_request}' -vv --http1.1 2>&1 |grep -E '(< HTTP|> bytes)'
< HTTP/1.1 200 OK
> bytes sent: 7969

Output of uname -a or ver

Linux 4.18.0-477.10.1.el8_8.x86_64 #1 SMP Wed Apr 5 13:35:01 EDT 2023 x86_64 x86_64 x86_64 GNU/Linux

Output of java -version

openjdk version "11.0.19" 2023-04-18 LTS

GraalVM version (if different from Java)

No response

Quarkus version or git rev

3.0.4.Final

Build tool (ie. output of mvnw --version or gradlew --version)

Apache Maven 3.8.8 (4c87b05d9aedce574290d1acc98575ed5eb6cd39)

Additional information

No response

[quarkus-bot]

/cc @geoand (openshift), @iocanel (openshift)

[geoand]

@cescoffier @vietj is this expected behavior?

[vietj]

it might be a change / protection added in Netty ? can we get a vertx reproducer ?

[floone]

Reproducer: http : https://github.com/floone/vertx-reproduce431

Based on io.vertx.example.core.http2.simple.Server

[cescoffier]

@floone Thanks for the reproducer!

[vietj]

what used to be the vertx version that makes it pass ?

[vietj]

it is related to HTTP/2 settings which controls what the client can send (the default value is 8192), if I modify the reproducer and use:

        Vertx.vertx().createHttpServer(new HttpServerOptions()
                .setUseAlpn(true)
                .setSsl(true)
                .setInitialSettings(new Http2Settings().setMaxHeaderListSize(16000))
                .setPemKeyCertOptions(new PemKeyCertOptions()
                        .setKeyValue(Buffer.buffer(key))
                        .setCertValue(Buffer.buffer(cert))
                )
        ).requestHandler(req -> req.response().end("Hello World!\n")).listen(8443);

Then I get 200 response.

It might be due to this change eclipse-vertx/vert.x@ed9db7f in 4.2.3

You might want to configure this value with an higher value, perhaps Vert.x default value should be increased too.

[floone]

Thank you for the investigation @vietj!

From a user perspective, we did not opt-in http/2, and we did not configure it explicitly so far. However, it is enabled by default in Quarkus [1], and our issue popped up because on our new openshift cluster, http/2 is enabled while it was not enabled on the old environment.

There is a section about http limits in the docs [2], but not specifically for http/2. I don't know (yet) how Quarkus and vertx play together, but I would have assumed that certain values are propagated downstream? I.e. if I set quarkus.http.limits.max-header-size (which defaults to 20k) I would have assumed it would hold for both http/1.1 and http/2, given the docs just state that http/2 is opt-out.

Is this just a documentation issue?

From my perspective, it would make sense when the default of 20k would hold for all http flavours.

What do you think?

[1] https://quarkus.io/guides/http-reference#http2-support
[2] https://quarkus.io/guides/http-reference#http-limits-configuration

[cescoffier]

Yes, HTTP/2 is now enabled on Openshift (finally!).

So, yes, it's a documentation issue. I will have a look.

[floone]

@cescoffier Thank you, much appreciated. If I understand correctly, the default values still differ (20k vs. 8k), but at least it is documented and can be configured consistently 👍

[cescoffier]

Yes, we cannot reuse the value because it's not really the same thing. But now you can configure them.

[cdennison]

This config isn't working for me https://quarkus.io/guides/http-reference#http-limits-configuration (Quarkus version 3.4).

This is what I had to do to add a custom option.

https://quarkus.io/guides/rest-client-reactive#use-custom-http-options

import io.vertx.core.http.HttpClientOptions;
import jakarta.ws.rs.ext.ContextResolver;
import jakarta.ws.rs.ext.Provider;
import org.eclipse.microprofile.config.inject.ConfigProperty;

@Provider
public class CustomHttpClientOptions implements ContextResolver<HttpClientOptions> {

  @ConfigProperty(name = "http.maxHeaderSize")
  private Integer maxHeaderSize;

  @Override
  public HttpClientOptions getContext(Class<?> type) {
    HttpClientOptions options = new HttpClientOptions();
    options.setMaxHeaderSize(maxHeaderSize);
    return options;
  }
}

[cescoffier]

Can you open another issue? Something might be wrong in the config processing.