roachtest: backup/2TB/n10cpu4 failed

[cockroach-teamcity]

roachtest.backup/2TB/n10cpu4 failed with artifacts on release-21.2 @ 09c8f05a9cb713b8bddffa03d0e322e8f2caf73b:

		  | 	/usr/local/go/src/runtime/asm_amd64.s:1571
		Wraps: (2) output in run_071954.869448900_n1_cockroach_sql
		Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-6767304-1664862744-20-n10cpu4:1 -- ./cockroach sql --insecure -e "
		  | 				BACKUP bank.bank TO 'gs://cockroachdb-backup-testing/teamcity-6767304-1664862744-20-n10cpu4?AUTH=implicit'" returned
		  | stderr:
		  | ERROR: failed to run backup: exporting 1826 ranges: googleapi: got HTTP response code 503 with body: Service Unavailable
		  | Failed running "sql"
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ``````
		  |   | ./cockroach sql --insecure -e "
		  |   | 				BACKUP bank.bank TO 'gs://cockroachdb-backup-testing/teamcity-6767304-1664862744-20-n10cpu4?AUTH=implicit'"
		  |   | ``````
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError

	monitor.go:128,backup.go:286,test_runner.go:777: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:116
		  | main.(*monitorImpl).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:124
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerBackup.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/backup.go:286
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:81
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:6222
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:233
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1571
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Reproduce

See: roachtest README

Same failure on other branches

• roachtest: backup/2TB/n10cpu4 failed [HTTP 503 error] #89057 roachtest: backup/2TB/n10cpu4 failed [C-test-failure O-roachtest O-robot T-disaster-recovery branch-master]
• roachtest: backup/2TB/n10cpu4 failed #88900 roachtest: backup/2TB/n10cpu4 failed [C-test-failure O-roachtest O-robot T-disaster-recovery branch-release-22.1 release-blocker]
• roachtest: backup/2TB/n10cpu4 failed #80030 roachtest: backup/2TB/n10cpu4 failed [C-test-failure O-roachtest O-robot T-disaster-recovery]

/cc @cockroachdb/disaster-recovery

This test on roachdash | Improve this report!

Jira issue: CRDB-20180

[msbutler]

http 503 error. removing release blocker.

[msbutler]

tracking this error on #89057