net/http: frequent HTTP2 INTERNAL_ERROR errors during module zip download since 2021-10-06 #51323

bcmills · 2022-02-22T19:04:49Z

#!watchflakes
post <- log ~ `golang\.org/[^@]+@v\d+\.\d+\.\d+[a-z0-9-.]*:.* stream error: stream ID \d+; INTERNAL_ERROR; received from peer`

The builders are seeing a lot of INTERNAL_ERROR results since 2021-10-06, typically for google.golang.org/protobuf but sometimes for other golang.org paths as well.

This may be related to #50541, which saw an uptick in errors from go.googlesource.com in about the same timeframe.

CC @golang/release

../../../../pkg/mod/cloud.google.com/go/[email protected]/client.go:25:2: google.golang.org/[email protected]: stream error: stream ID 3; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/[email protected]/internal/trace/trace.go:23:2: google.golang.org/[email protected]: stream error: stream ID 3; INTERNAL_ERROR; received from peer
../../../../pkg/mod/google.golang.org/[email protected]/status/status.go:34:2: google.golang.org/[email protected]: stream error: stream ID 3; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/[email protected]/save.go:26:2: google.golang.org/[email protected]: stream error: stream ID 3; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/[email protected]/cloudbuild/apiv1/v2/cloud_build_client.go:33:2: google.golang.org/[email protected]: stream error: stream ID 3; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/[email protected]/longrunning/autogen/operations_client.go:31:2: google.golang.org/[email protected]: stream error: stream ID 3; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/[email protected]/iam/iam.go:30:2: google.golang.org/[email protected]: stream error: stream ID 3; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/[email protected]/iam.go:24:2: google.golang.org/[email protected]: stream error: stream ID 3; INTERNAL_ERROR; received from peer

greplogs --dashboard -md -l -e 'golang\.org/[^@]+@v\d+\.\d+\.\d+[a-z0-9-.]*: stream error: stream ID \d+; INTERNAL_ERROR; received from peer' --since=2021-01-01

2022-02-19T16:23:54-3962a08-0261fa6/dragonfly-amd64
2022-02-10T22:36:44-59536be-0a6cf87/linux-amd64-wsl
2022-02-09T18:09:24-9862752-6a70ee2/darwin-arm64-11_0-toothrot
2022-02-09T16:29:28-dad3315-0a6cf87/darwin-arm64-12_0-toothrot
2022-02-09T16:29:28-095f870-0a6cf87/darwin-arm64-12_0-toothrot
2022-02-08T17:01:10-9b156ee-ef06a5f/plan9-amd64-0intro
2022-02-08T16:42:58-ac99473-e2277c8/darwin-arm64-12_0-toothrot
2022-02-03T19:55:40-cd36cc0-896df42/plan9-amd64-0intro
2022-01-25T22:56:45-c20fd7c-6eb58cd/plan9-amd64-0intro
2022-01-24T21:27:33-6944b10-671e115/plan9-amd64-0intro
2022-01-24T21:27:33-5e0467b-671e115/plan9-amd64-0intro
2022-01-14T16:41:18-4bd3f69-b41185c/plan9-386-0intro
2022-01-07T02:32:39-e83268e-2bb7f6b/linux-amd64-wsl
2022-01-05T22:15:55-43762dc-a845a56/darwin-arm64-12_0-toothrot
2022-01-03T23:45:12-be6af36-95b240b/plan9-amd64-0intro
2022-01-02T14:27:43-6944b10-c886143/plan9-amd64-0intro
2021-12-15T20:26:19-be6af36-07ed86c/darwin-arm64-12_0-toothrot
2021-12-14T18:18:45-ecdc095-becaeea/plan9-arm
2021-12-14T01:48:22-6944b10-1afa432/plan9-amd64-0intro
2021-11-10T19:35:55-03971e3-b954f58/darwin-arm64-11_0-toothrot
2021-10-28T14:25:03-103d89b-a3bb28e/darwin-amd64-10_15
2021-10-26T11:58:05-d418f37-1e2820a/openbsd-arm64-jsing
2021-10-14T15:16:55-a66eb64-ad99d88/linux-amd64-wsl
2021-10-14T01:51:22-e13a265-276fb27/windows-arm64-10
2021-10-13T00:11:47-281050f-4fb2e1c/darwin-amd64-11_0
2021-10-11T20:46:14-089bfa5-7023535/darwin-amd64-11_0
2021-10-07T20:37:43-86761ae-ef2ebbe/darwin-arm64-11_0-toothrot
2021-10-07T20:37:43-59d4e92-ef2ebbe/darwin-arm64-11_0-toothrot
2021-10-06T18:44:56-40a54f1-195945a/darwin-amd64-nocgo

The text was updated successfully, but these errors were encountered:

dmitshur · 2022-02-22T19:26:06Z

From logs, it's also happened for github.com dependencies such as github.com/yuin/goldmark in 2022-02-08T16:42:58-ac99473-e2277c8/darwin-arm64-12_0-toothrot.

Downloading happens with GOPROXY=https://proxy.golang.org set, so perhaps the go.googlesource.com and golang.org servers aren't relevant (they just happen to be where most of dependencies are coming from).

I've learned that INTERNAL_ERROR refers to an HTTP/2 error code that is described as "Implementation fault", but it's not clear to me if the server or client is at fault. CC @neild.

olix0r · 2022-03-21T19:24:03Z

Recently, we've encountered this fairly frequently (a few times per day) in GitHub Actions, usually for github.com URLs. For example:

#11 39.00 go mod download: github.com/coreos/[email protected]+incompatible: stream error: stream ID 4057; INTERNAL_ERROR; received from peer

heschi · 2022-03-21T20:56:43Z

The best explanation I have for this is some kind of infrequent incompatibility between the Go HTTP libraries and the Google front end servers.

bcmills · 2022-03-30T21:40:11Z

It occurs to me that this could be related to #49645 (CC @neild), although the timeline doesn't quite match up because the fix for that landed Nov. 17.

greplogs --dashboard -md -l -e 'stream error: stream ID \d+; INTERNAL_ERROR; received from peer' --since=2022-02-23 --omit=plan9

2022-03-29T15:43:06-e96d8cf-ae9ce82/windows-amd64-longtest
2022-03-23T17:12:00-87a8611-dd211cf/freebsd-arm64-dmgk

mintunitish · 2022-04-08T08:24:54Z

Is there any update on this issue! Experiencing too frequently

bcmills · 2022-05-06T13:56:58Z

greplogs -l -e 'stream error: stream ID \d+; INTERNAL_ERROR; received from peer' --since=2022-03-30 --omit=plan9
2022-05-05T16:31:21-15f5199-fdb640b/darwin-arm64-12_0-toothrot
2022-04-26T16:21:18-7b82a4e-e357634/solaris-amd64-oraclerel
2022-04-23T05:38:56-84e58bf-9717e8f/linux-mips64le-mengzhuo
2022-04-12T22:05:26-2897e13-d85694a/windows-amd64-longtest
2022-04-04T04:01:17-a8550c1-a92ca51/linux-mips64le-mengzhuo
2022-03-31T14:57:35-27fe37a-378221b/linux-amd64-longtest
2022-03-31T14:51:17-27fe37a-2ea9376/linux-amd64-longtest

bcmills · 2022-05-18T13:02:56Z

greplogs -l -e 'stream error: stream ID \d+; INTERNAL_ERROR; received from peer' --omit=plan9 --since=2022-05-05
2022-05-18T01:02:54-96df98a-5f2fdbe/windows-amd64-longtest
2022-05-17T21:28:43-405b155-1170771/linux-ppc64le-buildlet
2022-05-17T21:15:15-44db4eb-cf5fa2b/linux-ppc64le-power9osu
2022-05-17T20:59:58-b2400d8-8953302/linux-ppc64le-power9osu
2022-05-17T20:59:58-92f7ca4-8953302/linux-amd64-longtest
2022-05-17T20:51:53-85d78b3-b2ae2f5/linux-ppc64le-power9osu
2022-05-17T20:18:25-e5f449a-7607888/linux-ppc64le-power9osu
2022-05-17T19:40:11-e5f449a-d23c80b/linux-ppc64-buildlet
2022-05-17T18:19:34-85d78b3-076039a/linux-ppc64-buildlet
2022-05-17T18:19:34-84e58bf-076039a/linux-ppc64le-buildlet
2022-05-17T03:26:28-b2400d8-41b9d8c/linux-ppc64-buildlet
2022-05-17T01:50:34-1e55371-aedf298/linux-ppc64-buildlet
2022-05-16T20:19:45-403b017-2540f4e/aix-ppc64
2022-05-16T16:15:58-7413a13-db875f4/darwin-arm64-11_0-toothrot
2022-05-12T21:45:28-4aa4d2e-6365efb/linux-amd64-longtest
2022-05-05T16:31:21-15f5199-fdb640b/darwin-arm64-12_0-toothrot

bcmills · 2022-05-19T13:41:16Z

This could possibly be a duplicate of #36759 (reported as a regression Jan. 2020).

bcmills · 2022-05-23T19:09:15Z

Assigning to @neild (net/http owner) on the theory that this may be reproducing a previously-reported net/http client bug (#36759).

neild · 2022-06-09T20:33:36Z

stream error: stream ID N; INTERNAL_ERROR; received from peer means an HTTP/2 stream (a request) was closed by the remote endpoint with the error code INTERNAL_ERROR.

There are various other error codes to tell the client that it is misbehaving, such as PROTOCOL_ERROR. INTERNAL_ERROR is supposed to indicate an internal error on the remote side.

So whatever is doing the module zip download is claiming that it's unhealthy. It's not inconceivable that this is somehow the result of a net/http client bug, but if it is the remote side is doing a very bad job of telling us about it.

Maybe if we can reproduce the failure with GODEBUG=http2debug=2 enabled we might see something that helps indicate the source of the problem, but at the moment my suspicion is that the problem lies on the remote end of the connection.

neild · 2022-06-09T20:39:20Z

googleapis/google-cloud-go#784 seems like the same issue. From 2017, eventual "fix" was to notice INTERNAL_ERROR and retry the read.

heschi · 2022-06-09T21:06:38Z

The other end of the connection here is the GFEs; should we try to escalate to them internally?

neild · 2022-06-09T21:11:53Z

Yeah, I think so.

The possibilities I see are:

The net/http client is conjuring a peer-originated error out of thin air. This seems incredibly unlikely. I can find only one code path that adds "received from peer" and I don't see any way for it to lie.
The net/http client is misbehaving (very easy to believe) and the GFE is somehow responding with INTERNAL_ERROR. That's still a GFE bug, even if it's our fault for tickling it.
The net/http client is doing everything right, and the problem is entirely at or beyond the GFE.

At the same time, perhaps we should have a retry loop on module downloads. Connections break for various other reasons; doesn't seem like it would hurt to retry when they do.

Tomperez98 · 2022-06-17T13:49:01Z

Any idea how to solve this? Getting the following error when running go install golang.org/x/tools/gopls@latest.

/hpath/to/go/pkg/mod/golang.org/x/[email protected]/internal/lsp/source/hover.go:23:2: golang.org/x/[email protected]: stream error: stream ID 9; INTERNAL_ERROR; received from peer

ianlancetaylor · 2022-06-24T20:35:44Z

@neild What is the current status here? This issue is currently in the 1.19 milestone. Should it move to 1.20? To Backlog? Thanks.

neild · 2022-06-24T21:05:36Z

Our current belief is that the root cause is the GFE sending an INTERNAL_ERROR, probably due to the backend behind the GFE breaking the response stream.

This is not an net/http bug; net/http seems to be accurately reporting the error code sent by the GFE. (The error message could perhaps be clearer, but that's a secondary issue.)

Whatever tools are using net/http to fetch large files should be more robust in the handling of stream errors. There are many reasons a large fetch might break mid-stream, and it's unlikely we can entirely address this situation entirely by reducing the number of broken streams. This means we should add some form of retry to module zip downloads.

I think it's okay to move this to the 1.20 milestone.

`handleEvent` was handling the an `error` type by printing something, but then not continuing to raise the error. This means that in a stream error, particuarly of type INTERNAL_ERROR, the backoff/retry mechanism never took place as the return was `nil` and the agent just stopped. The problem is particularly relevant with Google Load Balancers and their handling of http2/websocket long lived connections: golang/go#51323 --------- Co-authored-by: Louis Garman <[email protected]>

gopherbot · 2023-09-19T14:18:10Z

Found new dashboard test flakes for:

#!watchflakes
post <- log ~ `golang\.org/[^@]+@v\d+\.\d+\.\d+[a-z0-9-.]*:.* stream error: stream ID \d+; INTERNAL_ERROR; received from peer`

2023-08-06 13:51 linux-riscv64-unmatched telemetry@6b1207f8 go@460dc37c x/telemetry/godev/internal/storage [build] (log)

internal/storage/storage_test.go:14:2: github.com/fullstorydev/emulators/[email protected]: read "https://goproxy.cn/@v/v0.0.0-20230523204811-eccb7d2267b0.mod": stream error: stream ID 29; INTERNAL_ERROR; received from peer

2023-08-28 22:24 linux-riscv64-unmatched pkgsite@ad6c6394 go@626a3cd9 (log)

../../../../pkg/mod/github.com/googleapis/gax-go/[email protected]/apierror/apierror.go:41:2: google.golang.org/[email protected]: read "https:/goproxy.cn/@v/v1.43.0.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/github.com/googleapis/gax-go/[email protected]/call_option.go:36:2: google.golang.org/[email protected]: read "https:/goproxy.cn/@v/v1.43.0.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/github.com/googleapis/gax-go/[email protected]/call_option.go:37:2: google.golang.org/[email protected]: read "https:/goproxy.cn/@v/v1.43.0.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/[email protected]/plugin/ocgrpc/client_stats_handler.go:23:2: google.golang.org/[email protected]: read "https:/goproxy.cn/@v/v1.43.0.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/[email protected]/plugin/ocgrpc/trace_common.go:22:2: google.golang.org/[email protected]: read "https:/goproxy.cn/@v/v1.43.0.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/[email protected]/plugin/ocgrpc/client.go:21:2: google.golang.org/[email protected]: read "https:/goproxy.cn/@v/v1.43.0.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/google.golang.org/[email protected]/transport/grpc/dial.go:31:2: google.golang.org/[email protected]: read "https:/goproxy.cn/@v/v1.43.0.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/google.golang.org/[email protected]/transport/grpc/dial.go:26:2: google.golang.org/[email protected]: read "https:/goproxy.cn/@v/v1.43.0.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/google.golang.org/[email protected]/transport/grpc/dial.go:27:2: google.golang.org/[email protected]: read "https:/goproxy.cn/@v/v1.43.0.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/google.golang.org/[email protected]/transport/grpc/dial.go:28:2: google.golang.org/[email protected]: read "https:/goproxy.cn/@v/v1.43.0.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer

2023-08-30 14:09 darwin-amd64-11_0 build@762ea330 go@a8d45a72 (log)

2023-08-30 16:14 netbsd-arm-bsiegert telemetry@03a44786 go@06df3292 (log)

internal/content/esbuild.go:14:2: github.com/evanw/[email protected]: read "https:/proxy.golang.org/@v/v0.17.19.zip": stream error: stream ID 5; INTERNAL_ERROR; received from peer

2023-09-10 18:25 netbsd-arm64-bsiegert crypto@3f0842a4 go@d110d7c4 (log)

— watchflakes

gopherbot · 2023-09-20T01:42:29Z

Found new dashboard test flakes for:

#!watchflakes
post <- log ~ `golang\.org/[^@]+@v\d+\.\d+\.\d+[a-z0-9-.]*:.* stream error: stream ID \d+; INTERNAL_ERROR; received from peer`

2023-09-19 17:00 darwin-amd64-13 build@c8fdba98 go@2b462646 [build] (log)

../../../../pkg/mod/cloud.google.com/go/[email protected]/apiv1/iampb/iam_policy.pb.go:28:2: google.golang.org/[email protected]: read "https:/proxy.golang.org/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/[email protected]/apiv1/iampb/policy.pb.go:27:2: google.golang.org/[email protected]: read "https:/proxy.golang.org/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/google.golang.org/[email protected]/status/status.go:35:2: google.golang.org/[email protected]: read "https:/proxy.golang.org/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/github.com/googleapis/gax-go/[email protected]/apierror/internal/proto/error.pb.go:27:2: google.golang.org/[email protected]: read "https:/proxy.golang.org/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/github.com/googleapis/gax-go/[email protected]/apierror/apierror.go:45:2: google.golang.org/[email protected]: read "https:/proxy.golang.org/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/[email protected]/internal/apiv2/stubs/storage.pb.go:30:2: google.golang.org/[email protected]: read "https:/proxy.golang.org/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/[email protected]/snapshot.go:24:2: google.golang.org/[email protected]: read "https:/proxy.golang.org/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/[email protected]/client.go:27:2: google.golang.org/[email protected]: read "https:/proxy.golang.org/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/[email protected]/save.go:27:2: google.golang.org/[email protected]: read "https:/proxy.golang.org/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/[email protected]/apiv1beta1/error_group_client.go:35:2: google.golang.org/[email protected]: read "https:/proxy.golang.org/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/[email protected]/apiv1/secretmanagerpb/service.pb.go:29:2: google.golang.org/[email protected]: read "https:/proxy.golang.org/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
internal/secret/flag.go:16:2: google.golang.org/[email protected]: read "https:/proxy.golang.org/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/[email protected]/apiv3/v2/monitoringpb/notification.pb.go:27:2: google.golang.org/[email protected]: read "https:/proxy.golang.org/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/[email protected]/apiv3/v2/monitoringpb/common.pb.go:27:2: google.golang.org/[email protected]: read "https:/proxy.golang.org/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/[email protected]/apiv3/v2/monitoringpb/metric.pb.go:27:2: google.golang.org/[email protected]: read "https:/proxy.golang.org/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/[email protected]/apiv3/v2/monitoringpb/metric.pb.go:28:2: google.golang.org/[email protected]: read "https:/proxy.golang.org/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/[email protected]/apiv3/v2/monitoringpb/group_service.pb.go:29:2: google.golang.org/[email protected]: read "https:/proxy.golang.org/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/[email protected]/apiv3/v2/monitoringpb/service.pb.go:28:2: google.golang.org/[email protected]: read "https:/proxy.golang.org/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/contrib.go.opencensus.io/exporter/[email protected]/monitoredresource/gcp/gcp_metadata_config.go:26:2: google.golang.org/[email protected]: read "https:/proxy.golang.org/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/contrib.go.opencensus.io/exporter/[email protected]/trace.go:28:2: google.golang.org/[email protected]: read "https:/proxy.golang.org/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/contrib.go.opencensus.io/exporter/[email protected]/metrics.go:36:2: google.golang.org/[email protected]: read "https:/proxy.golang.org/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/[email protected]/privateca/apiv1/certificate_authority_client.go:41:2: google.golang.org/[email protected]: read "https:/proxy.golang.org/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/[email protected]/apiv1/v2/cloudbuildpb/cloudbuild.pb.go:30:2: google.golang.org/[email protected]: read "https:/proxy.golang.org/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer

— watchflakes

gopherbot · 2023-09-26T15:44:30Z

Found new dashboard test flakes for:

#!watchflakes
post <- log ~ `golang\.org/[^@]+@v\d+\.\d+\.\d+[a-z0-9-.]*:.* stream error: stream ID \d+; INTERNAL_ERROR; received from peer`

2023-09-08 20:30 linux-riscv64-unmatched pkgsite@8025830b go@bda5e6c3 (log)

../../../../pkg/mod/github.com/googleapis/gax-go/[email protected]/apierror/internal/proto/error.pb.go:27:2: google.golang.org/[email protected]: read "https:/goproxy.cn/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 57; INTERNAL_ERROR; received from peer
../../../../pkg/mod/github.com/googleapis/gax-go/[email protected]/apierror/apierror.go:45:2: google.golang.org/[email protected]: read "https:/goproxy.cn/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 57; INTERNAL_ERROR; received from peer
../../../../pkg/mod/google.golang.org/[email protected]/status/status.go:35:2: google.golang.org/[email protected]: read "https:/goproxy.cn/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 57; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/[email protected]/profiler.go:59:2: google.golang.org/[email protected]: read "https:/goproxy.cn/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 57; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/[email protected]/apiv1beta1/error_group_client.go:35:2: google.golang.org/[email protected]: read "https:/goproxy.cn/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 57; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/[email protected]/apiv2/loggingpb/log_entry.pb.go:27:2: google.golang.org/[email protected]: read "https:/goproxy.cn/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 57; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/[email protected]/apiv2/loggingpb/logging_metrics.pb.go:29:2: google.golang.org/[email protected]: read "https:/goproxy.cn/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 57; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/[email protected]/apiv2/loggingpb/logging_metrics.pb.go:30:2: google.golang.org/[email protected]: read "https:/goproxy.cn/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 57; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/[email protected]/apiv2/loggingpb/log_entry.pb.go:28:2: google.golang.org/[email protected]: read "https:/goproxy.cn/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 57; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/[email protected]/apiv2/loggingpb/log_entry.pb.go:29:2: google.golang.org/[email protected]: read "https:/goproxy.cn/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 57; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/[email protected]/apiv2/loggingpb/logging_config.pb.go:29:2: google.golang.org/[email protected]: read "https:/goproxy.cn/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 57; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/[email protected]/apiv1/iampb/policy.pb.go:27:2: google.golang.org/[email protected]: read "https:/goproxy.cn/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 57; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/[email protected]/internal/apiv2/stubs/storage.pb.go:29:2: google.golang.org/[email protected]: read "https:/goproxy.cn/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 57; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/[email protected]/internal/apiv2/stubs/storage.pb.go:30:2: google.golang.org/[email protected]: read "https:/goproxy.cn/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 57; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/[email protected]/apiv3/v2/monitoringpb/notification.pb.go:27:2: google.golang.org/[email protected]: read "https:/goproxy.cn/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 57; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/[email protected]/apiv3/v2/monitoringpb/metric.pb.go:27:2: google.golang.org/[email protected]: read "https:/goproxy.cn/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 57; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/[email protected]/apiv3/v2/monitoringpb/service.pb.go:28:2: google.golang.org/[email protected]: read "https:/goproxy.cn/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 57; INTERNAL_ERROR; received from peer
../../../../pkg/mod/contrib.go.opencensus.io/exporter/[email protected]/monitoredresource/gcp/gcp_metadata_config.go:26:2: google.golang.org/[email protected]: read "https:/goproxy.cn/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 57; INTERNAL_ERROR; received from peer
../../../../pkg/mod/contrib.go.opencensus.io/exporter/[email protected]/trace.go:28:2: google.golang.org/[email protected]: read "https:/goproxy.cn/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 57; INTERNAL_ERROR; received from peer
../../../../pkg/mod/contrib.go.opencensus.io/exporter/[email protected]/metrics.go:36:2: google.golang.org/[email protected]: read "https:/goproxy.cn/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 57; INTERNAL_ERROR; received from peer

2023-09-19 20:58 darwin-amd64-nocgo pkgsite@14233af4 go@bf5d8c02 (log)

../../../../pkg/mod/google.golang.org/[email protected]/server.go:36:2: golang.org/x/[email protected]: read "https:/proxy.golang.org/@v/v0.15.0.zip": stream error: stream ID 31; INTERNAL_ERROR; received from peer
../../../../pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:31:2: golang.org/x/[email protected]: read "https:/proxy.golang.org/@v/v0.15.0.zip": stream error: stream ID 31; INTERNAL_ERROR; received from peer
../../../../pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:32:2: golang.org/x/[email protected]: read "https:/proxy.golang.org/@v/v0.15.0.zip": stream error: stream ID 31; INTERNAL_ERROR; received from peer
../../../../pkg/mod/google.golang.org/[email protected]/appengine.go:15:2: golang.org/x/[email protected]: read "https:/proxy.golang.org/@v/v0.15.0.zip": stream error: stream ID 31; INTERNAL_ERROR; received from peer
internal/source/source.go:34:2: golang.org/x/[email protected]: read "https:/proxy.golang.org/@v/v0.15.0.zip": stream error: stream ID 31; INTERNAL_ERROR; received from peer
../../../../pkg/mod/github.com/microcosm-cc/[email protected]/sanitize.go:41:2: golang.org/x/[email protected]: read "https:/proxy.golang.org/@v/v0.15.0.zip": stream error: stream ID 31; INTERNAL_ERROR; received from peer
internal/frontend/overview.go:16:2: golang.org/x/[email protected]: read "https:/proxy.golang.org/@v/v0.15.0.zip": stream error: stream ID 31; INTERNAL_ERROR; received from peer

— watchflakes

gopherbot · 2023-10-01T21:44:51Z

Change https://go.dev/cl/532035 mentions this issue: cmd/go/internal/modfetch: show real URL in response body read errors

gopherbot · 2023-10-04T19:55:28Z

Found new dashboard test flakes for:

#!watchflakes
post <- log ~ `golang\.org/[^@]+@v\d+\.\d+\.\d+[a-z0-9-.]*:.* stream error: stream ID \d+; INTERNAL_ERROR; received from peer`

2023-09-25 14:46 linux-s390x-ibm benchmarks@1082dde5 go@e47cab13 (log)

../../../../pkg/mod/golang.org/x/[email protected]/idna/idna10.0.0.go:26:2: golang.org/x/[email protected]: read "https:/proxy.golang.org/@v/v0.13.0.zip": stream error: stream ID 185; INTERNAL_ERROR; received from peer
../../../../pkg/mod/golang.org/x/[email protected]/idna/idna10.0.0.go:27:2: golang.org/x/[email protected]: read "https:/proxy.golang.org/@v/v0.13.0.zip": stream error: stream ID 185; INTERNAL_ERROR; received from peer
../../../../pkg/mod/golang.org/x/[email protected]/idna/idna10.0.0.go:28:2: golang.org/x/[email protected]: read "https:/proxy.golang.org/@v/v0.13.0.zip": stream error: stream ID 185; INTERNAL_ERROR; received from peer
../../../../pkg/mod/gitlab.com/golang-commonmark/[email protected]/charset.go:10:2: golang.org/x/[email protected]: read "https:/proxy.golang.org/@v/v0.13.0.zip": stream error: stream ID 185; INTERNAL_ERROR; received from peer

— watchflakes

gopherbot · 2023-10-27T11:15:03Z

Found new dashboard test flakes for:

#!watchflakes
post <- log ~ `golang\.org/[^@]+@v\d+\.\d+\.\d+[a-z0-9-.]*:.* stream error: stream ID \d+; INTERNAL_ERROR; received from peer`

2023-10-25 13:59 netbsd-arm-bsiegert telemetry@5ab6b58d go@0046c141 (log)

../../../../../pkg/mod/golang.org/x/[email protected]/idna/idna10.0.0.go:26:2: golang.org/x/[email protected]: read "https:/proxy.golang.org/@v/v0.9.0.zip": stream error: stream ID 99; INTERNAL_ERROR; received from peer
../../../../../pkg/mod/golang.org/x/[email protected]/idna/idna10.0.0.go:27:2: golang.org/x/[email protected]: read "https:/proxy.golang.org/@v/v0.9.0.zip": stream error: stream ID 99; INTERNAL_ERROR; received from peer
../../../../../pkg/mod/golang.org/x/[email protected]/idna/idna10.0.0.go:28:2: golang.org/x/[email protected]: read "https:/proxy.golang.org/@v/v0.9.0.zip": stream error: stream ID 99; INTERNAL_ERROR; received from peer

— watchflakes

CL 233437 added a redactedURL field to proxyRepo, a struct that already had a field named 'url'. Neither fields were documented, so the similar names suggest the most natural interpretation that proxyRepo.redactedURL is equivalent to proxyRepo.url.Redacted() rather than something else. That's possibly why it was joined with the module version in CL 406675. It turns out the two URLs differ in more than just redaction: one is the base proxy URL with (escaped) module path joined, the other is just the base proxy URL, in redacted form. Document and rename the fields to make the distinction more clear, and include all 3 of base module proxy URL + module path + module version in the reported URL, rather than just the first and third bits as seen in the errors at https://go.dev/issue/51323#issuecomment-1735812250. For #51323. Updates #38680. Updates #52727. Change-Id: Ib4b134b548adeec826ee88fe51a2cf580fde0516 Cq-Include-Trybots: luci.golang.try:gotip-linux-amd64-longtest Reviewed-on: https://go-review.googlesource.com/c/go/+/532035 LUCI-TryBot-Result: Go LUCI <[email protected]> Reviewed-by: Bryan Mills <[email protected]> Auto-Submit: Dmitri Shuralyov <[email protected]> Reviewed-by: Dmitri Shuralyov <[email protected]>

gopherbot · 2023-11-15T02:57:57Z

Found new dashboard test flakes for:

#!watchflakes
post <- log ~ `golang\.org/[^@]+@v\d+\.\d+\.\d+[a-z0-9-.]*:.* stream error: stream ID \d+; INTERNAL_ERROR; received from peer`

2023-11-10 21:13 solaris-amd64-oraclerel pkgsite@545ce2ad go@31f0af1e (log)

../../../../pkg/mod/golang.org/x/[email protected]/idna/idna10.0.0.go:25:2: golang.org/x/[email protected]: read "https://proxy.golang.org/golang.org/x/text/@v/v0.14.0.zip": stream error: stream ID 109; INTERNAL_ERROR; received from peer
../../../../pkg/mod/golang.org/x/[email protected]/idna/idna10.0.0.go:26:2: golang.org/x/[email protected]: read "https://proxy.golang.org/golang.org/x/text/@v/v0.14.0.zip": stream error: stream ID 109; INTERNAL_ERROR; received from peer
../../../../pkg/mod/golang.org/x/[email protected]/idna/idna10.0.0.go:27:2: golang.org/x/[email protected]: read "https://proxy.golang.org/golang.org/x/text/@v/v0.14.0.zip": stream error: stream ID 109; INTERNAL_ERROR; received from peer
../../../../pkg/mod/github.com/google/[email protected]/html.go:14:2: golang.org/x/[email protected]: read "https://proxy.golang.org/golang.org/x/text/@v/v0.14.0.zip": stream error: stream ID 109; INTERNAL_ERROR; received from peer
../../../../pkg/mod/github.com/jackc/[email protected]/auth_scram.go:27:2: golang.org/x/[email protected]: read "https://proxy.golang.org/golang.org/x/text/@v/v0.14.0.zip": stream error: stream ID 109; INTERNAL_ERROR; received from peer
internal/godoc/dochtml/dochtml.go:32:2: golang.org/x/[email protected]: read "https://proxy.golang.org/golang.org/x/text/@v/v0.14.0.zip": stream error: stream ID 109; INTERNAL_ERROR; received from peer
internal/godoc/dochtml/dochtml.go:33:2: golang.org/x/[email protected]: read "https://proxy.golang.org/golang.org/x/text/@v/v0.14.0.zip": stream error: stream ID 109; INTERNAL_ERROR; received from peer
internal/frontend/imports.go:16:2: golang.org/x/[email protected]: read "https://proxy.golang.org/golang.org/x/text/@v/v0.14.0.zip": stream error: stream ID 109; INTERNAL_ERROR; received from peer

— watchflakes

gopherbot · 2023-11-19T19:45:55Z

Found new dashboard test flakes for:

#!watchflakes
post <- log ~ `golang\.org/[^@]+@v\d+\.\d+\.\d+[a-z0-9-.]*:.* stream error: stream ID \d+; INTERNAL_ERROR; received from peer`

2023-11-14 16:31 linux-riscv64-unmatched telemetry@69313e64 go@1c15291f (log)

../../../../../pkg/mod/cloud.google.com/go/[email protected]/apiv1/iampb/iam_policy.pb.go:28:2: google.golang.org/[email protected]: read "https://goproxy.cn/google.golang.org/genproto/@v/v0.0.0-20230410155749-daa745c078e1.zip": stream error: stream ID 35; INTERNAL_ERROR; received from peer
../../../../../pkg/mod/cloud.google.com/go/[email protected]/apiv1/iampb/policy.pb.go:27:2: google.golang.org/[email protected]: read "https://goproxy.cn/google.golang.org/genproto/@v/v0.0.0-20230410155749-daa745c078e1.zip": stream error: stream ID 35; INTERNAL_ERROR; received from peer
../../../../../pkg/mod/google.golang.org/[email protected]/status/status.go:35:2: google.golang.org/[email protected]: read "https://goproxy.cn/google.golang.org/genproto/@v/v0.0.0-20230410155749-daa745c078e1.zip": stream error: stream ID 35; INTERNAL_ERROR; received from peer
../../../../../pkg/mod/github.com/googleapis/gax-go/[email protected]/apierror/internal/proto/error.pb.go:27:2: google.golang.org/[email protected]: read "https://goproxy.cn/google.golang.org/genproto/@v/v0.0.0-20230410155749-daa745c078e1.zip": stream error: stream ID 35; INTERNAL_ERROR; received from peer
../../../../../pkg/mod/github.com/googleapis/gax-go/[email protected]/apierror/apierror.go:41:2: google.golang.org/[email protected]: read "https://goproxy.cn/google.golang.org/genproto/@v/v0.0.0-20230410155749-daa745c078e1.zip": stream error: stream ID 35; INTERNAL_ERROR; received from peer
../../../../../pkg/mod/cloud.google.com/go/[email protected]/internal/apiv2/stubs/storage.pb.go:30:2: google.golang.org/[email protected]: read "https://goproxy.cn/google.golang.org/genproto/@v/v0.0.0-20230410155749-daa745c078e1.zip": stream error: stream ID 35; INTERNAL_ERROR; received from peer

— watchflakes

gopherbot · 2023-11-21T20:17:57Z

Found new dashboard test flakes for:

#!watchflakes
post <- log ~ `golang\.org/[^@]+@v\d+\.\d+\.\d+[a-z0-9-.]*:.* stream error: stream ID \d+; INTERNAL_ERROR; received from peer`

2023-11-21 19:04 windows-amd64-longtest go@ebca52ee cmd/go.TestScript (log)

vcs-test.golang.org rerouted to http://127.0.0.1:54795
https://vcs-test.golang.org rerouted to https://127.0.0.1:54796
go test proxy running at GOPROXY=http://127.0.0.1:54797/mod
--- FAIL: TestScript (0.14s)
    --- FAIL: TestScript/mod_sumdb_file_path (91.40s)
        script_test.go:132: 2023-11-21T19:43:03Z
        script_test.go:134: $WORK=C:\Users\gopher\AppData\Local\Temp\1\cmd-go-test-1865174972\tmpdir2564125123\mod_sumdb_file_path3035018066
        script_test.go:156: 
            > [!net:proxy.golang.org] skip
            [condition not met]
...
            go: golang.org/x/[email protected]: reading file:///C:%5CUsers%5Cgopher%5CAppData%5CLocal%5CTemp%5C1%5Ccmd-go-test-1865174972%5Ctmpdir2564125123%5Cmod_sumdb_file_path3035018066/sumproxy/golang.org/x/text/@v/v0.3.2.mod: The system cannot find the path specified.
            [exit status 1]
            > [GOOS:windows] env GOPROXY=file:///$WORK/sumproxy,https://proxy.golang.org
            > [!GOOS:windows] env GOPROXY=file://$WORK/sumproxy,https://proxy.golang.org
            [condition not met]
            > go get golang.org/x/[email protected]
            [stderr]
            go: downloading golang.org/x/text v0.3.2
            go: golang.org/x/[email protected]: read "https://proxy.golang.org/golang.org/x/text/@v/v0.3.2.zip": stream error: stream ID 11; INTERNAL_ERROR; received from peer
        script_test.go:156: FAIL: testdata\script\mod_sumdb_file_path.txt:46: go get golang.org/x/[email protected]: exit status 1

2023-11-21 19:26 linux-386-longtest go@b3b44244 cmd/go.TestScript (log)

vcs-test.golang.org rerouted to http://127.0.0.1:41933
https://vcs-test.golang.org rerouted to https://127.0.0.1:38623
go test proxy running at GOPROXY=http://127.0.0.1:41757/mod
2023/11/21 19:54:24 http: TLS handshake error from 127.0.0.1:56838: EOF
2023/11/21 19:54:24 http: TLS handshake error from 127.0.0.1:56852: EOF
2023/11/21 19:55:20 http: TLS handshake error from 127.0.0.1:36984: read tcp 127.0.0.1:38623->127.0.0.1:36984: read: connection reset by peer
--- FAIL: TestScript (0.05s)
    --- FAIL: TestScript/mod_sumdb_file_path (2.36s)
        script_test.go:132: 2023-11-21T19:54:20Z
        script_test.go:134: $WORK=/workdir/tmp/cmd-go-test-791329614/tmpdir2447884709/mod_sumdb_file_path3863367325
...
            > [GOOS:windows] env GOPROXY=file:///$WORK/sumproxy,https://proxy.golang.org
            [condition not met]
            > [!GOOS:windows] env GOPROXY=file://$WORK/sumproxy,https://proxy.golang.org
            > ! go get golang.org/x/[email protected]
            [stderr]
            go: downloading golang.org/x/text v0.3.2
            go: golang.org/x/[email protected]: read "https://proxy.golang.org/golang.org/x/text/@v/v0.3.2.zip": stream error: stream ID 7; INTERNAL_ERROR; received from peer
            [exit status 1]
            > stderr '^go: golang.org/x/[email protected]: verifying module: golang.org/x/[email protected]: reading file://.*/sumdb/sum.golang.org/lookup/golang.org/x/[email protected]: (no such file or directory|.*cannot find the path specified.*)'
        script_test.go:156: FAIL: testdata/script/mod_sumdb_file_path.txt:18: stderr '^go: golang.org/x/[email protected]: verifying module: golang.org/x/[email protected]: reading file://.*/sumdb/sum.golang.org/lookup/golang.org/x/[email protected]: (no such file or directory|.*cannot find the path specified.*)': no match for `(?m)^go: golang.org/x/[email protected]: verifying module: golang.org/x/[email protected]: reading file://.*/sumdb/sum.golang.org/lookup/golang.org/x/[email protected]: (no such file or directory|.*cannot find the path specified.*)` in stderr

— watchflakes

gopherbot · 2023-11-27T20:53:34Z

Found new dashboard test flakes for:

#!watchflakes
post <- log ~ `golang\.org/[^@]+@v\d+\.\d+\.\d+[a-z0-9-.]*:.* stream error: stream ID \d+; INTERNAL_ERROR; received from peer`

2023-11-27 16:49 darwin-amd64-longtest go@6317fb41 cmd/go.TestScript (log)

vcs-test.golang.org rerouted to http://127.0.0.1:49949
https://vcs-test.golang.org rerouted to https://127.0.0.1:49950
go test proxy running at GOPROXY=http://127.0.0.1:49951/mod
--- FAIL: TestScript (0.45s)
    --- FAIL: TestScript/mod_sumdb_file_path (29.25s)
        script_test.go:132: 2023-11-27T17:59:36Z
        script_test.go:134: $WORK=/tmp/buildlet/tmp/cmd-go-test-4127345601/tmpdir881471550/mod_sumdb_file_path2607082938
        script_test.go:156: 
            > [!net:proxy.golang.org] skip
            [condition not met]
...
            go: golang.org/x/[email protected]: reading file:///tmp/buildlet/tmp/cmd-go-test-4127345601/tmpdir881471550/mod_sumdb_file_path2607082938/sumproxy/golang.org/x/text/@v/v0.3.2.mod: no such file or directory
            [exit status 1]
            > [GOOS:windows] env GOPROXY=file:///$WORK/sumproxy,https://proxy.golang.org
            [condition not met]
            > [!GOOS:windows] env GOPROXY=file://$WORK/sumproxy,https://proxy.golang.org
            > go get golang.org/x/[email protected]
            [stderr]
            go: downloading golang.org/x/text v0.3.2
            go: golang.org/x/[email protected]: read "https://proxy.golang.org/golang.org/x/text/@v/v0.3.2.zip": stream error: stream ID 11; INTERNAL_ERROR; received from peer
        script_test.go:156: FAIL: testdata/script/mod_sumdb_file_path.txt:46: go get golang.org/x/[email protected]: exit status 1

— watchflakes

gopherbot · 2023-11-30T17:15:20Z

Found new dashboard test flakes for:

#!watchflakes
post <- log ~ `golang\.org/[^@]+@v\d+\.\d+\.\d+[a-z0-9-.]*:.* stream error: stream ID \d+; INTERNAL_ERROR; received from peer`

2023-11-30 03:00 linux-riscv64-unmatched pkgsite@9f20944f go@b4fa5b16 (log)

internal/postgres/delete.go:11:2: github.com/Masterminds/[email protected]: read "https://goproxy.cn/github.com/%21masterminds/squirrel/@v/v1.5.2.zip": stream error: stream ID 169; INTERNAL_ERROR; received from peer

— watchflakes

gopherbot · 2023-12-01T02:16:03Z

Found new dashboard test flakes for:

#!watchflakes
post <- log ~ `golang\.org/[^@]+@v\d+\.\d+\.\d+[a-z0-9-.]*:.* stream error: stream ID \d+; INTERNAL_ERROR; received from peer`

2023-11-27 18:10 netbsd-arm64-bsiegert perf@b5375226 go@bda1ef13 (log)

benchseries/chart.go:22:2: gonum.org/v1/[email protected]: read "https://proxy.golang.org/gonum.org/v1/plot/@v/v0.10.1.zip": stream error: stream ID 23; INTERNAL_ERROR; received from peer
benchseries/chart.go:23:2: gonum.org/v1/[email protected]: read "https://proxy.golang.org/gonum.org/v1/plot/@v/v0.10.1.zip": stream error: stream ID 23; INTERNAL_ERROR; received from peer
benchseries/chart.go:24:2: gonum.org/v1/[email protected]: read "https://proxy.golang.org/gonum.org/v1/plot/@v/v0.10.1.zip": stream error: stream ID 23; INTERNAL_ERROR; received from peer
benchseries/chart.go:16:2: gonum.org/v1/[email protected]: read "https://proxy.golang.org/gonum.org/v1/plot/@v/v0.10.1.zip": stream error: stream ID 23; INTERNAL_ERROR; received from peer
benchseries/chart.go:17:2: gonum.org/v1/[email protected]: read "https://proxy.golang.org/gonum.org/v1/plot/@v/v0.10.1.zip": stream error: stream ID 23; INTERNAL_ERROR; received from peer

— watchflakes

gopherbot · 2024-03-07T11:35:45Z

Found new dashboard test flakes for:

#!watchflakes
post <- log ~ `golang\.org/[^@]+@v\d+\.\d+\.\d+[a-z0-9-.]*:.* stream error: stream ID \d+; INTERNAL_ERROR; received from peer`

2024-03-06 21:06 linux-riscv64-unmatched telemetry@d5a85b27 go@90d24b4f (log)

../../../../../pkg/mod/github.com/googleapis/gax-go/[email protected]/apierror/apierror.go:44:2: google.golang.org/[email protected]: read "https://goproxy.cn/google.golang.org/api/@v/v0.149.0.zip": stream error: stream ID 25; INTERNAL_ERROR; received from peer
../../../../../pkg/mod/cloud.google.com/go/[email protected]/internal/apiv2/storage_client.go:30:2: google.golang.org/[email protected]: read "https://goproxy.cn/google.golang.org/api/@v/v0.149.0.zip": stream error: stream ID 25; INTERNAL_ERROR; received from peer
../../../../../pkg/mod/cloud.google.com/go/[email protected]/internal/apiv2/doc.go:91:2: google.golang.org/[email protected]: read "https://goproxy.cn/google.golang.org/api/@v/v0.149.0.zip": stream error: stream ID 25; INTERNAL_ERROR; received from peer
../../../../../pkg/mod/cloud.google.com/go/[email protected]/internal/apiv2/storage_client.go:32:2: google.golang.org/[email protected]: read "https://goproxy.cn/google.golang.org/api/@v/v0.149.0.zip": stream error: stream ID 25; INTERNAL_ERROR; received from peer
../../../../../pkg/mod/cloud.google.com/go/[email protected]/internal/apiv2/storage_client.go:33:2: google.golang.org/[email protected]: read "https://goproxy.cn/google.golang.org/api/@v/v0.149.0.zip": stream error: stream ID 25; INTERNAL_ERROR; received from peer
../../../../../pkg/mod/cloud.google.com/go/[email protected]/bucket.go:32:2: google.golang.org/[email protected]: read "https://goproxy.cn/google.golang.org/api/@v/v0.149.0.zip": stream error: stream ID 25; INTERNAL_ERROR; received from peer
../../../../../pkg/mod/cloud.google.com/go/[email protected]/acl.go:24:2: google.golang.org/[email protected]: read "https://goproxy.cn/google.golang.org/api/@v/v0.149.0.zip": stream error: stream ID 25; INTERNAL_ERROR; received from peer
../../../../../pkg/mod/cloud.google.com/go/[email protected]/http_client.go:41:2: google.golang.org/[email protected]: read "https://goproxy.cn/google.golang.org/api/@v/v0.149.0.zip": stream error: stream ID 25; INTERNAL_ERROR; received from peer
../../../../../pkg/mod/cloud.google.com/go/[email protected]/http_client.go:42:2: google.golang.org/[email protected]: read "https://goproxy.cn/google.golang.org/api/@v/v0.149.0.zip": stream error: stream ID 25; INTERNAL_ERROR; received from peer

— watchflakes

Sebastin-cr · 2024-03-11T09:23:03Z

I am facing the below errors
http2: Framer 0xc00029b5e0: read RST_STREAM stream=3 len=4 ErrCode=INTERNAL_ERROR
http2: Transport received RST_STREAM stream=3 len=4 ErrCode=INTERNAL_ERROR
RoundTrip failure: stream error: stream ID 3; INTERNAL_ERROR; received from peer

Any root cause for this?

neild · 2024-03-11T18:49:04Z

In stream error: stream ID 3; INTERNAL_ERROR; received from peer, the "received from peer" indicates that the INTERNAL_ERROR code was received from the remote endpoint.

That is, the server you are talking to sent back an INTERNAL_ERROR code, presumably indicating that it has encountered an internal error of some form. net/http is just relaying that fact to you. The root cause is presumably on the server side.

Sebastin-cr · 2024-03-12T10:08:50Z

is server sending RST_STREAM incase of any internal error introduced in latest go version or was it existing in earlier versions because when we were using x/net/v0.0.0-20211029224645-99673261e6eb this behaviour was not coming and when we updated it to x/net/v0.19 for fortify fixes,we are encountering this error @neild

neild · 2024-03-18T22:59:58Z

I'm sorry, but I don't have enough details to attempt to answer that question.

The error is coming from the remote server. Did you update your server to a newer version of x/net, your client, or both? If it's just the client, then you want to ask why your server is producing INTERNAL_ERROR errors, not why the client is surfacing them. If you did update the server, then I'd really need a reproduction case to attempt to debug the issue (and we should do that on a new issue, not this one, which is not about errors being produced by Go servers).

v0.0.0-20211029224645-99673261e6eb to v0.19.0 is two years of changes. I'm afraid I don't remember everything that happened in that time.

gopherbot added this to the Unreleased milestone Feb 22, 2022

bcmills added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Feb 22, 2022

dmitshur changed the title ~~x/website: frequent HTTP2 INTERNAL_ERROR errors from *.golang.org since 2021-10-06~~ net/http, x/net/http2, proxy.golang.org: frequent HTTP2 INTERNAL_ERROR errors during module zip download since 2021-10-06 Feb 22, 2022

dmitshur mentioned this issue Mar 30, 2022

x/build/maintner/godata, x/build/maintner/maintnerd/maintapi: getGoData helper should handle failure without failing test, otherwise that causes occasional false positive test failures due to HTTP/2 internal errors while syncing segments #52048

Closed

bcmills mentioned this issue May 3, 2022

x/build/internal/gomote: response code: 502 in TestWriteFileFromURL on darwin/amd64 #52671

Closed

bcmills mentioned this issue May 6, 2022

build dashboard triage log #52653

Closed

bcmills mentioned this issue May 19, 2022

net/http: http.Client.Do() sometimes throws INTERNAL_ERROR when using high number of parallel HTTP/2 requests (regression between 1.12 and 1.13.4) #36759

Open

bcmills assigned neild May 23, 2022

bcmills changed the title ~~net/http, x/net/http2, proxy.golang.org: frequent HTTP2 INTERNAL_ERROR errors during module zip download since 2021-10-06~~ net/http: frequent HTTP2 INTERNAL_ERROR errors during module zip download since 2021-10-06 May 23, 2022

bcmills modified the milestones: Unreleased, Go1.19 May 23, 2022

joedian added this to Release Blockers Jun 15, 2022

ianlancetaylor modified the milestones: Go1.19, Go1.20 Jun 24, 2022

gopherbot added this to the Go1.22 milestone Aug 8, 2023

tomwardill-payoneer mentioned this issue Aug 11, 2023

Return error on stream error for retry leg100/otf#565

Merged

bcmills mentioned this issue Sep 19, 2023

go test -short -timeout=1h40m0s ./...: unrecognized failures #62717

Closed

bcmills modified the milestones: Go1.22, Backlog Sep 19, 2023

vikasmulaje mentioned this issue Oct 3, 2023

Preflight check container fails with Error: failed to extract tarball: stream error: stream ID 41; INTERNAL_ERROR; received from peer redhat-openshift-ecosystem/openshift-preflight#1072

Closed

DmitriyLewen mentioned this issue Nov 7, 2023

bug(image): INTERNAL_ERROR when scanning an image aquasecurity/trivy#5440

Open

zyxkad mentioned this issue Feb 23, 2024

下载突然卡住然后报错 LiterMC/go-openbmclapi#29

Closed

pk910 mentioned this issue Aug 26, 2024

workaround for eventstream disconnects ethpandaops/dora#116

Merged

gabyhelp mentioned this issue Sep 20, 2024

proxy.golang.org, cmd/go, net/http: http2 PROTOCOL_ERRORS from the proxy #69544

Closed

gabyhelp mentioned this issue Oct 9, 2024

proxy.golang.org: Intermittent TLS/Network errors with Google's Module Proxy #69826

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

net/http: frequent HTTP2 INTERNAL_ERROR errors during module zip download since 2021-10-06 #51323

net/http: frequent HTTP2 INTERNAL_ERROR errors during module zip download since 2021-10-06 #51323

bcmills commented Feb 22, 2022 •

edited

Loading

dmitshur commented Feb 22, 2022

olix0r commented Mar 21, 2022

heschi commented Mar 21, 2022

bcmills commented Mar 30, 2022

mintunitish commented Apr 8, 2022

bcmills commented May 6, 2022

bcmills commented May 18, 2022

bcmills commented May 19, 2022

bcmills commented May 23, 2022

neild commented Jun 9, 2022

neild commented Jun 9, 2022

heschi commented Jun 9, 2022

neild commented Jun 9, 2022

Tomperez98 commented Jun 17, 2022

ianlancetaylor commented Jun 24, 2022

neild commented Jun 24, 2022

gopherbot commented Sep 19, 2023

gopherbot commented Sep 20, 2023

gopherbot commented Sep 26, 2023

gopherbot commented Oct 1, 2023 •

edited by dmitshur

Loading

gopherbot commented Oct 4, 2023

gopherbot commented Oct 27, 2023

gopherbot commented Nov 15, 2023

gopherbot commented Nov 19, 2023

gopherbot commented Nov 21, 2023

gopherbot commented Nov 27, 2023

gopherbot commented Nov 30, 2023

gopherbot commented Dec 1, 2023

gopherbot commented Mar 7, 2024

Sebastin-cr commented Mar 11, 2024

neild commented Mar 11, 2024

Sebastin-cr commented Mar 12, 2024 •

edited

Loading

neild commented Mar 18, 2024

net/http: frequent HTTP2 INTERNAL_ERROR errors during module zip download since 2021-10-06 #51323

net/http: frequent HTTP2 INTERNAL_ERROR errors during module zip download since 2021-10-06 #51323

Comments

bcmills commented Feb 22, 2022 • edited Loading

dmitshur commented Feb 22, 2022

olix0r commented Mar 21, 2022

heschi commented Mar 21, 2022

bcmills commented Mar 30, 2022

mintunitish commented Apr 8, 2022

bcmills commented May 6, 2022

bcmills commented May 18, 2022

bcmills commented May 19, 2022

bcmills commented May 23, 2022

neild commented Jun 9, 2022

neild commented Jun 9, 2022

heschi commented Jun 9, 2022

neild commented Jun 9, 2022

Tomperez98 commented Jun 17, 2022

ianlancetaylor commented Jun 24, 2022

neild commented Jun 24, 2022

gopherbot commented Sep 19, 2023

gopherbot commented Sep 20, 2023

gopherbot commented Sep 26, 2023

gopherbot commented Oct 1, 2023 • edited by dmitshur Loading

gopherbot commented Oct 4, 2023

gopherbot commented Oct 27, 2023

gopherbot commented Nov 15, 2023

gopherbot commented Nov 19, 2023

gopherbot commented Nov 21, 2023

gopherbot commented Nov 27, 2023

gopherbot commented Nov 30, 2023

gopherbot commented Dec 1, 2023

gopherbot commented Mar 7, 2024

Sebastin-cr commented Mar 11, 2024

neild commented Mar 11, 2024

Sebastin-cr commented Mar 12, 2024 • edited Loading

neild commented Mar 18, 2024

bcmills commented Feb 22, 2022 •

edited

Loading

gopherbot commented Oct 1, 2023 •

edited by dmitshur

Loading

Sebastin-cr commented Mar 12, 2024 •

edited

Loading