net/http: http client regression building with js/wasm and running on Chrome: net::ERR_H2_OR_QUIC_REQUIRED

[jnowls]

What version of Go are you using (go version)?

$ go version
go version go1.21.0 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=''
GOARCH='wasm'
GOBIN=''
GOCACHE='/home/h/.cache/go-build'
GOENV='/home/h/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/h/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='js'
GOPATH='/home/h/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/lib/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/lib/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.21.0'
GCCGO='gccgo'
GOWASM=''
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='0'
GOMOD='/home/h/code/pk/axesfull/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build4139731908=/tmp/go-build -gno-record-gcc-switches'
uname -sr: Linux 6.4.8-arch1-1
LSB Version:	n/a
Distributor ID:	Arch
Description:	Arch Linux
Release:	rolling
Codename:	n/a
/usr/lib/libc.so.6: GNU C Library (GNU libc) stable release version 2.38.

What did you do?

NOTE: I only get these errors in Chromium based browsers, not in Firefox.

I have this following function:

type Account struct {
	ID string `json:"id"`
}

func TestPersist(t *testing.T) {
	fmt.Printf("runtime.Version(): %v\n", runtime.Version())
	acc := Account{"5e7fd575-b5f5-4d7e-8fa7-70b5fef7662f"}
	buf, _ := json.Marshal(acc)
	r := bytes.NewBuffer(buf)
	req, err := http.NewRequest("POST", "https://nk.axesfull.dev/v2/account/authenticate/device", r)
	if err != nil {
		t.Fatalf("could not create request: %v", err)
	}
	_, err = http.DefaultClient.Do(req)
	if err != nil {
		t.Fatalf("could not authenticate account: %v", err)
	}
}

compiled with:
GOOS=js GOARCH=wasm go test -c -o test.wasm -run TestPersist

I ran this with the default wasm_exec.html thats provided in $GOROOT/misc/wasm with just adding test.v to argv:.

		async function run() {
			console.clear();
			await go.run(inst);
			go.arvc = ["js", "-test.v"]
			inst = await WebAssembly.instantiate(mod, go.importObject); // reset instance
		}

This works with go1.20, but not with go1.21.

Reproducible example available at https://github.com/jnowls/wasm-http-chrome - just run ./run.sh

What did you expect to see?

The browser successfully complete the request (as seen in Firefox and older versions of chrome)

What did you see instead?

Firefox go1.21 and go1.20:

Chrome go1.21 and go1.20:

[mknyszek]

CC @neild

[samstride]

Hi,

Running into this issue with go v1.21.0.

go version go1.21.0 linux/amd64

No issues on go v1.20.7

My understanding is that this has something to do with HTTP 1 vs 2.

In my case, since I am on local machine, the server runs on localhost (http1) and hence run into this issue. If the connection is over HTTP2 (PROD server), I do not get this error.

Will be nice to see feature detection added as described here?

https://developer.chrome.com/articles/fetch-streaming-requests/#doesnt-work-on-http1x

Thanks.

[jnowls]

Hi,

Running into this issue with go v1.21.0.

go version go1.21.0 linux/amd64

No issues on go v1.20.7

My understanding is that this has something to do with the connection not being secure.

In my case, since I am on local machine, the server runs on localhost and hence run into this issue. If the connection is secure (PROD server), I do not get this error.

Will be nice to see feature detection added as described here?

https://developer.chrome.com/articles/fetch-streaming-requests/#doesnt-work-on-http1x

Thanks.

Just to clarify, based on that article and my own testing, it's not an issue with HTTP or HTTPS, but more with HTTP 1, regardless of whether SSL/TLS is there or not.

[haruyama480]

I reproduced the same error as reported.

And, I found that the following merge was the trigger.
https://go-review.googlesource.com/c/go/+/458395

The following line implements that request body is streamed if the browser supports streaming, regardless fetch() use http/1 or 2.
https://go-review.googlesource.com/c/go/+/458395/10/src/net/http/roundtrip_js.go#135

The error net::ERR_H2_OR_QUIC_REQUIRED literally indicates that the streaming is not supported by http/1.
If roundtrip_js.go doesn't need to support http/2, 458395 should be reverted. Otherwise, it seems to need a flag of whether client uses http1 or 2.

supportsPostRequestStreams determines whether the browser can stream request body. Firefox doesn't support it, but Chrome does support it. So, this error doesn't happen on Firefox. I guess that all POST requests from almost Chrome via http/1 fail until fix.

[haruyama480]

I wrote minimum example. https://github.com/haruyama480/go1210-request-streaming-failed

[samstride]

@jnowls. Correct. Sorry, should have worded it better.

Just to clarify, based on that article and my own testing, it's not an issue with HTTP or HTTPS, but more with HTTP 1, regardless of whether SSL/TLS is there or not.

[haruyama480]

Let me summarize this issue.

Issue

• Excuting wasm binary built by go1.21.0 which does POST request to 'http://..' on in Chromium-based browser caused the error net::ERR_H2_OR_QUIC_REQUIRED
• this issue doesn't reproduce for go1.20, Firefox, or GET request.

Cause

• This issue is triggered by the commit 458395. The commit intended enable request streaming if available in client. However, according to this article, request streaming isn't supported by http/2, and doesn't work if server or proxy doesn't support it.
• Although Chrome doesn't implement http/2 requests for no TLS, Chrome does streaming requests to 'http://..' nevertheless via http/1, and consequently failed even sending requests.

How to solve

I think there are following solutions.
a. implement an option so that clients can control the use of stream
b. greedy use streaming for https request and disable it for http request
c. both
d. or just revert it, and wait utill enough discussion

a and b have limitations: a forces developers to use options properly and b ignores server or proxy supportings. but, a would be easy way, if you implement it via global vars, such as omitBundledHTTP2 though it makes developer use only one of http or https on one wasm binary.

Yes, I'm not very familiar with the go repository, so could I ask how to fix to Go team or code owner? @johanbrandhorst

If there is chance, I would like to contribute to fix it. :)

[johanbrandhorst]

CC @hawkinsw. Please feel free to open a CL with a fix, sorry for the inconvenience!

[hhhapz]

Thanks for the offer of contributing the fix @haruyama480 :)

I don't see any quick and easy way to determine whether or not the browser will complete a HTTP request with H2 or H1 before it happens :(. Just disabling request streaming for HTTP also probably isn't a proper fix since POST requests with HTTPS+H1 (what OP describes) also fail.

If we do end up going with implementing it such that clients select whether to use the streaming API, that should probably not be done with a global variable, it will only make things more complicated to deal with, though I'm not sure what the best alternative to give the client the control is.

I personally think request streaming should be reverted for now, and revisited at a later point where the streaming API provides proper mechanisms to fallback in case of no support from the server.

[haruyama480]

@hhhapz
Thanks for the thoughtful advice!

I don't see any quick and easy way to determine whether or not the browser will complete a HTTP request with H2 or H1 before it happens :(.

I completely agree. b ignores HTTPS+H1, and isn't the proper way.

Also, I agree that adding a global variable isn't good. But, I'm wondering if nethttpomithttp2 tag can be an opt-out flag for developers to avoid H2. This is easy to implement and makes sense.

Falling back will be a good idea. Chaining fetch promise properly would be a general solution. But I'm not sure the nice way to handle req.Body and ReadableStream twice for H2 and H1. That may be complicated.

I think It would be enough to focus only on the opt-out flag and decide how to implement it in this issue.

[hhhapz]

Any particular reason to make this opt-out instead of opt-in? In the long term lots of users who will be unaware of this will be confused why their code isn't working (or worse, will be unaware it doesn't work in certain contexts). Making it opt-in sounds like the better solution since there is no automatic fallback for this feature at the moment, whereas the regular API is going to work without concern in all contexts.

Ultimately, while this is a nice feature, it's not a showbreaker (it just came out this release so very few people are using it, and it's mainly a performance boost ). Until a proper and robust solution is found, where multiple requests don't need to be done, and the developer never needs to care about whether request streaming is happening or not, this feature probably shouldn't be used.

[johanbrandhorst]

I agree that it should either be turned off entirely or opt-in, so that it always works as expected by default. The change that introduced this was https://go-review.googlesource.com/c/go/+/458395, so the first step would be to revert this (so that it's working in tip) and then consider reintroducing it with an opt-in.

[samstride]

@johanbrandhorst , any idea which version of Go the fix will be released in? Thanks.

[johanbrandhorst]

At the earliest I expect this to be available in 1.22 unfortunately. It's not severe enough to be included in a patch release (I don't think, at least). It will also depend on when someone can help contribute the fixes outlined.

[samstride]

@johanbrandhorst , so the revert and the opt-in will need to go in together? Any chance of doing the revert as a patch and the opti-in later?

[hhhapz]

At the earliest I expect this to be available in 1.22 unfortunately. It's not severe enough to be included in a patch release (I don't think, at least). It will also depend on when someone can help contribute the fixes outlined.

Really disappointed if this is the case :/. I have a core interactive application that uses WASM. A revert being included in a patch release should be reconsidered, as this completely breaks the usability of POST requests in Chrome over HTTP, especially since there is no workaround for this fix at all (apart from "just use HTTP/2")

I am not familiar with the specific process of how changes are determined to be included within patches vs minor releases, but until this is fixed I am unable to upgrade my application or use any 1.21 features that I would like (including many other improvements to WASM).

[haruyama480]

Any particular reason to make this opt-out instead of opt-in?

depend on finding a good fallback.

If there is a good fallback mechanism, opt-out would be ideally better than opt-in because Chrome user take good performance without configuration. Otherwise opt-in would be better. However, I suspect opt-in mechanism is to be deleted when we find the mechanism later, so I don't feel need to implement opt-in for now.

the first step would be to revert this

I agree that.

I think it should be released as a patch version up because the latest minor version(means max(1.21.*)) should work by the default, as @hhhapz pointed.

[haruyama480]

I see nobody disagree about revert, and submitted CL of revert!
https://go-review.googlesource.com/c/go/+/522955

[gopherbot]

Change https://go.dev/cl/522955 mentions this issue: net/http: revert "support streaming POST content in wasm"

[neild]

If I understand correctly, https://go.dev/cl/458395 which added support for streaming HTTP request bodies in wasm bulids, results requests failing when:

1. The request has a body.
2. The program is being executed by Chrome.
3. The server handling the request supports HTTP1, but neither HTTP/2 nor HTTP/3.

Also, it sounds like there is no workaround: Requests always fail when the above conditions are met.

Is that all correct?

If so, I agree that we should roll this change back. We should also backport the rollback to the next 1.21 minor release, as this is both a significant regression (you can't send requests at all under these circumstances) and there is no workaround.

[johanbrandhorst]

Thanks for chiming in Neil, I'm pleased to hear that this will be considered for the next minor release. Given that @haruyama480 has submitted a CL to remove this from tip, what is the process for backporting the fix to the release branch? Is it automated or will it need someone to cherry pick the revert to the release branch? Do we need a backport issue?

[neild]

@gopherbot please backport to 1.21. This is a regression with no workaround.

[gopherbot]

Backport issue(s) opened: #62328 (for 1.21).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://go.dev/wiki/MinorReleases.

[neild]

what is the process for backporting the fix to the release branch?

Process is documented here: https://go.dev/wiki/MinorReleases.

In short, the release manager will approve (or reject) the backport issue. If/when approved, I'll make a cherry-pick CL for the backport, and the release manager will merge it into the 1.21 release branch.

[hawkinsw]

Thank you all for the great conversation around this issue. I am sorry for a number of things:

1. For having caused a problem in the first place; and
2. For not having responded or joined the conversation earlier.

I thought lots about this problem over the weekend and I think that I might have a way to allow the user to opt-in: Some applications (like one that I am writing) explicitly call http2.ConfigureTransports after having created an http.Client (or explicitly use an http2.Transport when creating an http.Client in the first place).

In those cases, we could use a type assertion in the runtime to check whether the round tripper's transport is the H2 variety. Obviously, if the user has taken these actions to seek out H2, then we would reasonably assume that they would be okay if it failed in the scenario where a connection to the server could not be completed over that protocol.

What do people think about this as a possibility for doing opt in?

[johanbrandhorst]

I don't think adding explicit type assertions into the code is the way to go. The problem with that is that it becomes impossible for users to wrap the transport transparently. I'd sooner see we allow the user to be more explicit about opting in. We already have a pattern for configuring the HTTP request via headers (See https://go.googlesource.com/go/+/refs/heads/master/src/net/http/roundtrip_js.go#26 and friends). We could probably just add another header that would opt into the behavior?

[hawkinsw]

I don't think adding explicit type assertions into the code is the way to go. The problem with that is that it becomes impossible for users to wrap the transport transparently. I'd sooner see we allow the user to be more explicit about opting in. We already have a pattern for configuring the HTTP request via headers (See https://go.googlesource.com/go/+/refs/heads/master/src/net/http/roundtrip_js.go#26 and friends). We could probably just add another header that would opt into the behavior?

That is perfectly okay with me! I was just excited that I found a technical way to do what we wanted!

[gopherbot]

Change https://go.dev/cl/513458 mentions this issue: net/http: don't stream from a few common static io.Reader types on Wasm

[gopherbot]

Change https://go.dev/cl/524855 mentions this issue: [release-branch.go1.21] net/http: revert "support streaming POST content in wasm"