-
Notifications
You must be signed in to change notification settings - Fork 17.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
meta: high rate of 5XX errors from go.googlesource.com since 2021-09-08 #50541
Comments
We should also check to make sure these errors are accurately reflected in the monitoring for these services. (It isn't obvious to me that they are even showing up.) |
For some reason the failure rate to this service on the
2022-02-04T23:42:50-f1903fd/linux-386-longtest |
Another of what appears to be a “googlesource server flakiness” failure mode:
2022-04-11T16:31:42-910a33a/linux-amd64-longtest |
And more of the conventional flakiness too.
2022-04-11T16:31:43-036b615/linux-386-longtest |
|
@golang/release, I've filed this internally as b/231704574. |
|
Change https://go.dev/cl/405714 mentions this issue: |
Go tests don't include timestamps by default, but we would like to have them in order to correlate builder failures with server and network logs. Since many of the Go tests with external network and service dependencies are script tests for the 'go' command, logging timestamps here adds a lot of logging value with one simple and very low-risk change. For #50541. For #52490. For #52545. For #52851. Change-Id: If3fa86deb4a216ec6a1abc4e6f4ee9b05030a729 Reviewed-on: https://go-review.googlesource.com/c/go/+/405714 Reviewed-by: Dmitri Shuralyov <[email protected]> Auto-Submit: Bryan Mills <[email protected]> Reviewed-by: Dmitri Shuralyov <[email protected]> TryBot-Result: Gopher Robot <[email protected]> Run-TryBot: Bryan Mills <[email protected]>
Found new dashboard test flakes for:
2023-01-17 19:53 linux-arm64-longtest go@526b8956 cmd/go.TestScript (log)
2023-01-20 18:12 linux-386-longtest go@e38cb25b cmd/go/internal/modload.TestQueryImport (log)
|
Found new dashboard test flakes for:
2023-01-30 16:00 linux-amd64-longtest go@d42c08a2 cmd/go.TestScript (log)
2023-01-30 16:41 linux-amd64-longtest go@8c347a09 cmd/go.TestScript (log)
2023-01-30 18:44 linux-arm64-longtest go@d75a8673 cmd/go.TestScript (log)
2023-01-30 18:49 linux-386-longtest go@6ab721df cmd/go.TestScript (log)
2023-01-30 18:49 linux-amd64-longtest-race go@01b9656f cmd/go.TestScript (log)
2023-01-30 18:50 linux-arm64-longtest go@4c5d9799 cmd/go.TestScript (log)
2023-01-30 20:36 linux-amd64-longtest-race go@2ab0e046 cmd/go.TestScript (log)
2023-01-30 20:36 linux-arm64-longtest go@2ab0e046 cmd/go.TestScript (log)
2023-01-31 15:09 linux-amd64-longtest go@066b7805 cmd/go/internal/modfetch.TestCodeRepo (log)
2023-01-31 16:01 linux-amd64-longtest-race go@e1f9499e cmd/go.TestScript (log)
2023-01-31 16:53 linux-amd64-longtest go@43115ff0 cmd/go.TestScript (log)
2023-01-31 16:55 linux-arm64-longtest go@0067d814 cmd/go/internal/modload.TestQueryImport (log)
2023-01-31 17:38 linux-amd64-longtest-race go@adc1db23 cmd/go.TestScript (log)
|
Found new dashboard test flakes for:
2023-01-30 15:19 linux-amd64-longtest go@9e45b1d5 cmd/go.TestScript (log)
2023-01-30 18:10 linux-amd64-longtest go@dbfdc446 cmd/go.TestScript (log)
2023-01-31 13:55 linux-arm64-longtest go@da0c375c cmd/go.TestScript (log)
|
Found new dashboard test flakes for:
2023-02-27 17:33 linux-amd64-longtest go@2de406bb cmd/go.TestScript (log)
2023-03-01 17:58 linux-amd64-longtest-race go@cf3d0655 cmd/go.TestScript (log)
2023-03-01 21:23 linux-arm64-longtest go@09852e75 cmd/go.TestScript (log)
|
Found new dashboard test flakes for:
2023-03-13 14:16 linux-arm64-longtest go@b561ebab cmd/go.TestScript (log)
|
Found new dashboard test flakes for:
2023-03-22 21:13 linux-arm64-longtest go@65896f68 cmd/go.TestScript (log)
|
Found new dashboard test flakes for:
2023-03-22 20:51 linux-amd64-longtest-race go@d35dd190 cmd/go.TestScript (log)
|
Found new dashboard test flakes for:
2023-03-29 19:20 linux-amd64-longtest-race go@9a164d1c cmd/go.TestScript (log)
|
Found new dashboard test flakes for:
2023-04-14 17:01 linux-amd64-longtest-race go@ac9d777c cmd/go.TestScript (log)
|
Found new dashboard test flakes for:
2023-04-14 18:40 linux-386-longtest go@35ec948d cmd/go.TestScript (log)
|
Found new dashboard test flakes for:
2023-04-20 14:56 linux-arm64-longtest go@4f9a9669 cmd/go.TestScript (log)
|
Found new dashboard test flakes for:
2023-04-21 16:47 linux-arm64-longtest go@e23322e2 cmd/go.TestScript (log)
|
Found new dashboard test flakes for:
2023-04-24 14:06 linux-386-longtest go@ee42d468 cmd/go.TestScript (log)
|
Found new dashboard test flakes for:
2023-05-02 19:41 linux-amd64-longtest-race go@a32232cb cmd/go.TestScript (log)
|
Found new dashboard test flakes for:
2023-05-10 15:11 linux-arm64-longtest go@93d9035c cmd/go.TestScript (log)
|
Found new dashboard test flakes for:
2023-05-10 15:11 linux-amd64-longtest-race go@93d9035c cmd/go/internal/modload.TestQueryImport (log)
|
Found new dashboard test flakes for:
2023-05-10 20:35 linux-386-longtest go@200a01ff cmd/go.TestScript (log)
|
Found new dashboard test flakes for:
2023-05-10 20:35 linux-amd64-longtest-race go@200a01ff cmd/go.TestScript (log)
|
Found new dashboard test flakes for:
2023-05-12 13:51 linux-arm64-longtest go@c3db9af3 cmd/go.TestScript (log)
|
Found new dashboard test flakes for:
2023-05-17 13:57 linux-amd64-longtest go@54031132 cmd/go.TestScript (log)
|
Found new dashboard test flakes for:
2023-05-19 17:41 linux-amd64-longtest go@dc4993e7 cmd/go.TestScript (log)
|
Found new dashboard test flakes for:
2023-05-24 00:49 linux-386-longtest go@0c301519 cmd/go.TestScript (log)
|
Found new dashboard test flakes for:
2023-05-24 21:17 linux-amd64-longtest go@85e84a49 cmd/go.TestScript (log)
|
Found new dashboard test flakes for:
2023-05-24 22:39 linux-amd64-longtest go@79a8997a cmd/go.TestScript (log)
|
Found new dashboard test flakes for:
2023-05-25 17:51 linux-amd64-longtest go@ff07c540 cmd/go.TestScript (log)
|
Found new dashboard test flakes for:
2023-05-22 21:48 linux-386-longtest go@65cc8e6a cmd/go.TestScript (log)
|
Converting |
Found new dashboard test flakes for:
2023-07-20 23:09 linux-amd64-longtest-race go@a04a665a cmd/go.TestScript (log)
|
Found new dashboard test flakes for:
2023-08-15 14:52 linux-amd64-longtest-race go@b51a4dd6 cmd/go.TestScript (log)
|
Found new dashboard test flakes for:
2023-09-15 15:33 linux-amd64-longtest go@5f872db5 cmd/go.TestScript (log)
|
Some of the
cmd/go
long-mode tests end up cloning repos controlled by the Go project, such asgolang.org/x/sys
.Other long-mode tests send requests to a dedicated test server (
vcs-test.golang.org
), various GitHub repos, packages hosted onrsc.io
, andgopkg.in
. The failure rate againstvcs-test.golang.org
is, as far as I can tell, negligible. The GitHub andgopkg.in
tests for the most part check integration with services outside the control of the Go project. Thersc.io
tests have a low but nontrivial failure rate (filed as #49954), and could perhaps be migrated tovcs-test.golang.org
if need be.However, the dependency on
go.googlesource.com
for thex
repos is not easy to avoid, and as far as I can tell its failure rate since September has dwarfed all of those other services. Many of these failures already result in long (2m+) hangs that could otherwise push the test over its deadline, so I don't think it would be viable to just add retries (#28194) — especially given that if the problem could be fixed by retrying, the server could presumably do that retry internally itself.I would rather not lose coverage or add complexity to the tests by fetching the Go repos only through
proxy.golang.org
— @golang/release, is it viable to get a higher SLA ongo.googlesource.com
?greplogs --dashboard -md -l -e 'https://go\.googlesource\.com.* 5\d\d\b' --since=2021-01-01
2022-01-10T17:06:19-9e7ea35/linux-386-longtest
2022-01-05T21:22:03-2b39d86/linux-386-longtest
2021-12-08T15:30:52-9fe77de/linux-amd64-longtest
2021-12-03T21:23:11-b3e1fbf/linux-386-longtest
2021-12-02T23:34:26-5f65520/linux-386-longtest
2021-12-02T22:03:11-d3bf627/linux-386-longtest
2021-12-02T20:49:56-bbe1be5/linux-386-longtest
2021-12-02T20:45:47-3c6295d/linux-386-longtest
2021-12-01T17:13:34-08ecdf7/linux-386-longtest
2021-11-30T20:04:58-7ccbcc9/linux-386-longtest
2021-11-30T18:49:39-5f63f16/linux-amd64-longtest
2021-11-30T18:09:02-931d80e/linux-386-longtest
2021-11-30T18:09:02-931d80e/linux-amd64-longtest
2021-11-29T22:02:48-2f6d382/linux-386-longtest
2021-11-24T21:11:45-b77f5f9/linux-386-longtest
2021-11-24T21:09:36-7e5331a/linux-amd64-longtest
2021-11-23T19:31:05-0f64c21/linux-amd64-longtest
2021-11-19T21:36:17-0f75a9e/linux-amd64-longtest
2021-11-16T14:33:48-50dac3b/linux-386-longtest
2021-11-11T20:20:42-3e94140/linux-386-longtest
2021-11-11T13:58:28-d76b1ac/linux-386-longtest
2021-11-10T18:24:14-6406e09/linux-amd64-longtest
2021-11-09T22:58:24-578ada4/linux-386-longtest
2021-11-09T19:01:20-526b2ef/linux-386-longtest
2021-11-09T18:23:16-55e6e82/linux-amd64-longtest
2021-11-08T16:44:14-7ee3f14/linux-amd64-longtest
2021-11-05T17:47:28-6f32d20/linux-386-longtest
2021-11-05T00:52:04-1c4cfd8/linux-386-longtest
2021-11-04T21:52:36-8ad0a7e/linux-386-longtest
2021-11-04T14:17:18-901bf29/linux-386-longtest
2021-11-03T20:30:17-7f2463c/linux-amd64-longtest
2021-11-02T13:43:24-4d7bf41/linux-386-longtest
2021-11-01T19:42:30-5f1bcd1/linux-386-longtest
2021-11-01T15:55:29-0ae4c7f/linux-amd64-longtest
2021-10-29T19:35:47-5ec139f/linux-386-longtest
2021-10-29T18:56:29-6afdf01/linux-386-longtest
2021-10-29T18:35:59-5ec2427/linux-386-longtest
2021-10-29T18:35:20-9ac1ee2/linux-386-longtest
2021-10-29T18:23:03-413672f/linux-386-longtest
2021-10-28T20:27:44-5e21728/linux-amd64-longtest
2021-10-28T16:54:29-5fce1d9/linux-386-longtest
2021-10-28T01:15:26-b2fe2eb/linux-386-longtest
2021-10-27T21:37:54-749f6e9/linux-386-longtest
2021-10-27T20:24:48-a155a30/linux-386-longtest
2021-10-27T20:03:17-68bd512/linux-386-longtest
2021-10-27T19:34:49-ffd2284/linux-386-longtest
2021-10-27T19:18:20-514ebae/linux-386-longtest
2021-10-27T18:08:49-bb49eb3/linux-386-longtest
2021-10-26T20:12:32-f8779b9/linux-amd64-longtest
2021-10-26T17:10:45-76cef81/linux-amd64-longtest
2021-10-26T14:24:17-283d8a3/linux-386-longtest
2021-10-25T17:46:41-9012996/linux-amd64-longtest
2021-10-22T00:57:18-9ff91b9/linux-amd64-longtest
2021-10-21T19:08:43-b8da7e4/linux-386-longtest
2021-10-21T02:35:47-fca46d0/linux-386-longtest
2021-10-12T20:20:41-d032b2b/linux-386-longtest
2021-10-12T14:32:53-36a265a/linux-386-longtest
2021-10-05T19:28:36-7ae83c8/linux-386-longtest
2021-10-01T14:30:33-8d494b0/linux-386-longtest
2021-09-27T19:42:34-315cec2/linux-386-longtest
2021-09-21T14:02:52-7d67f8d/linux-amd64-longtest
2021-09-15T17:32:52-6196979/linux-386-longtest
2021-09-15T17:32:52-6196979/linux-amd64-longtest
2021-09-15T16:32:27-72bb818/linux-386-longtest
2021-09-15T16:32:27-72bb818/linux-amd64-longtest
2021-09-14T23:20:18-2933c45/linux-386-longtest
2021-09-14T23:07:15-137543b/linux-amd64-longtest
2021-09-14T23:03:28-3a72175/linux-amd64-longtest
2021-09-13T21:48:41-42057e9/linux-386-longtest
2021-09-13T20:42:44-81a4fe6/linux-386-longtest
2021-09-09T01:25:40-d4579c2-d62866e/darwin-arm64-11_0-toothrot
2021-09-08T16:59:47-d419f9c/linux-amd64-longtest
2021-09-08T16:19:36-409434d/linux-386-longtest
2021-05-26T22:43:54-1d5298d/linux-amd64-longtest
The text was updated successfully, but these errors were encountered: