Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: ssh_problem failed #112310

Open
cockroach-teamcity opened this issue Oct 13, 2023 · 71 comments
Open

roachtest: ssh_problem failed #112310

cockroach-teamcity opened this issue Oct 13, 2023 · 71 comments
Labels
branch-release-23.2 Used to mark GA and release blockers, technical advisories, and bugs for 23.2 O-roachtest O-robot Originated from a bot. T-testeng TestEng Team X-infra-flake the automatically generated issue was closed due to an infrastructure problem not a product issue
Milestone
23.2

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Oct 13, 2023 •
edited by cockroach-jira-scripts
Loading

roachtest.ssh_problem failed with artifacts on release-23.2 @ bd71370a4afb6cf2a7718a4f20ccba6a9436bdb6:

test cdc/workload/kv100/nodes=5/cpu=16/ranges=100/server=processor/protocol=mux/format=json/sink=null failed due to (cluster.go:2074).Start: one or more parallel execution failure(s): error persisted after 2 attempts: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
test artifacts and logs in: /artifacts/cdc/workload/kv100/nodes=5/cpu=16/ranges=100/server=processor/protocol=mux/format=json/sink=null/run_1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-32350
@cockroach-teamcity cockroach-teamcity added branch-release-23.2 Used to mark GA and release blockers, technical advisories, and bugs for 23.2 O-roachtest O-robot Originated from a bot. T-testeng TestEng Team X-infra-flake the automatically generated issue was closed due to an infrastructure problem not a product issue labels Oct 13, 2023
@cockroach-teamcity cockroach-teamcity added this to the 23.2 milestone Oct 13, 2023
@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-23.2 @ bd71370a4afb6cf2a7718a4f20ccba6a9436bdb6:

test unoptimized-query-oracle/disable-rules=half/seed-multi-region failed due to (cluster.go:2113).Stop: cluster.StopE: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
test artifacts and logs in: /artifacts/unoptimized-query-oracle/disable-rules=half/seed-multi-region/run_1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Oct 13, 2023
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-23.2 @ bd71370a4afb6cf2a7718a4f20ccba6a9436bdb6:

test tpccbench/nodes=3/cpu=16/mt-shared-process failed due to (cluster.go:2074).Start: one or more parallel execution failure(s): error persisted after 2 attempts: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=3/cpu=16/mt-shared-process/run_1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-23.2 @ dc24db6b7ed2f53d034ae9b8038ce08fc7d57542:

test tpcc/multiregion/survive=region/chaos=true failed due to (monitor.go:153).Wait: monitor failure: could not stop node :1-3: cluster.StopE: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
test artifacts and logs in: /artifacts/tpcc/multiregion/survive=region/chaos=true/run_1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=true , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

This was referenced Oct 30, 2023
Open
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-23.2 @ 348d11425a1184ce4de8a6f8dc85995cd3c653bc:

test import/tpcc/warehouses=4000/geo failed due to (cluster.go:2074).Start: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
test artifacts and logs in: /artifacts/import/tpcc/warehouses=4000/geo/run_1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=true , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-23.2 @ c702128fa39c4aa84e810b898064fd2d66ace765:

test failover/partial/lease-leader failed due to (cluster.go:2239).Run: full command output in run_171217.690767084_n5_sudo-iptables-A-INPU.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
(cluster.go:2239).Run: context canceled
test artifacts and logs in: /artifacts/failover/partial/lease-leader/run_1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=2 , ROACHTEST_encrypted=false , ROACHTEST_metamorphicBuild=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Nov 16, 2023
Closed
@cockroach-teamcity cockroach-teamcity mentioned this issue Dec 5, 2023
Open
@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-23.2 @ 9a13104643153a4233d626e0d398e4b7175674bd:

test cdc/workload/kv100/nodes=5/cpu=16/ranges=100/control failed due to (cluster.go:2046).Start: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
test artifacts and logs in: /artifacts/cdc/workload/kv100/nodes=5/cpu=16/ranges=100/control/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Jan 5, 2024
Closed
@cockroach-teamcity cockroach-teamcity mentioned this issue Jan 18, 2024
Closed
@cockroach-teamcity cockroach-teamcity mentioned this issue Feb 14, 2024
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-23.2 @ 895000ff72c7a40a075dca3ec4687028ebe468b9:

test failover/non-system/crash/lease=expiration failed: (cluster.go:2187).Start: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
test artifacts and logs in: /artifacts/failover/non-system/crash/lease=expiration/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=2
  • ROACHTEST_encrypted=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-23.2 @ 895000ff72c7a40a075dca3ec4687028ebe468b9:

test sysbench/oltp_delete/nodes=3/cpu=32/conc=256 failed: (monitor.go:153).Wait: monitor failure: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
test artifacts and logs in: /artifacts/sysbench/oltp_delete/nodes=3/cpu=32/conc=256/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=32
  • ROACHTEST_encrypted=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-23.2 @ 895000ff72c7a40a075dca3ec4687028ebe468b9:

test failover/non-system/blackhole/lease=expiration failed: (cluster.go:2301).Run: full command output in run_162818.375871235_n6_sudo-iptables-F.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
(cluster.go:2301).Run: context canceled
test artifacts and logs in: /artifacts/failover/non-system/blackhole/lease=expiration/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=2
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-23.2 @ 895000ff72c7a40a075dca3ec4687028ebe468b9:

test failover/non-system/blackhole-send/lease=expiration failed: (cluster.go:2301).Run: full command output in run_162823.829930868_n6_sudo-iptables-A-OUTP.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
(cluster.go:2301).Run: context canceled
test artifacts and logs in: /artifacts/failover/non-system/blackhole-send/lease=expiration/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=2
  • ROACHTEST_encrypted=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-23.2 @ 895000ff72c7a40a075dca3ec4687028ebe468b9:

test failover/non-system/blackhole-send failed: (cluster.go:2301).Run: full command output in run_162819.394431270_n5_sudo-iptables-F.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
(cluster.go:2301).Run: context canceled
test artifacts and logs in: /artifacts/failover/non-system/blackhole-send/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=2
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-23.2 @ 895000ff72c7a40a075dca3ec4687028ebe468b9:

test failover/non-system/crash failed: (cluster.go:2187).Start: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/failover/non-system/crash/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=2
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-23.2 @ 895000ff72c7a40a075dca3ec4687028ebe468b9:

test unoptimized-query-oracle/disable-rules=half/rand-tables failed: (cluster.go:2238).Stop: cluster.StopE: one or more parallel execution failure(s): error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
test artifacts and logs in: /artifacts/unoptimized-query-oracle/disable-rules=half/rand-tables/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-23.2 @ 895000ff72c7a40a075dca3ec4687028ebe468b9:

test failover/chaos/read-write failed: (cluster.go:2187).Start: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
(cluster.go:2301).Run: context canceled
(cluster.go:2301).Run: context canceled
(cluster.go:2301).Run: context canceled
(cluster.go:2301).Run: context canceled
test artifacts and logs in: /artifacts/failover/chaos/read-write/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=2
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-23.2 @ 895000ff72c7a40a075dca3ec4687028ebe468b9:

test failover/chaos/read-write/lease=expiration failed: (cluster.go:2265).Signal: cluster.Signal: one or more parallel execution failure(s): error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
(cluster.go:2301).Run: context canceled
(cluster.go:2301).Run: context canceled
(cluster.go:2301).Run: context canceled
(cluster.go:2301).Run: context canceled
test artifacts and logs in: /artifacts/failover/chaos/read-write/lease=expiration/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=2
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

This was referenced May 26, 2024
Closed
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-23.2 @ 2982a0aca59a1dd5aa771761f0e1fee2aa4e0212:

test failover/non-system/disk-stall failed: (cluster.go:2230).Start: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/failover/non-system/disk-stall/run_1/ssh/ssh_153555.758061424_n4_run-start-script.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
(cluster.go:2344).Run: context canceled
test artifacts and logs in: /artifacts/failover/non-system/disk-stall/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=2
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Jun 21, 2024
Closed
@cockroach-teamcity cockroach-teamcity mentioned this issue Jul 1, 2024
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-23.2 @ 7ca8340d8b1316144a9f0f53e736f168d99a0bab:

test tpccbench/nodes=3/cpu=16/mt-shared-process failed: (cluster.go:2230).Start: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa -i /home/roach/.ssh/google_compute_engine [email protected]:certs.tar /tmp/certs.tar1526602570
ssh: connect to host 18.220.246.252 port 22: Connection timed out
: exit status 1
test artifacts and logs in: /artifacts/tpccbench/nodes=3/cpu=16/mt-shared-process/cpu_arch=arm64/run_1

Parameters:

  • ROACHTEST_arch=arm64
  • ROACHTEST_cloud=aws
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

This test on roachdash | Improve this report!

This was referenced Jul 13, 2024
Open
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-23.2 @ e309a2fd7adba4454dbd65d319062e51c584845b:

test failover/partial/lease-leader/lease=expiration failed: (cluster.go:2230).Start: failed to find services to register: failed to find 1 open ports: TRANSIENT_ERROR(open_ports): output:: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/failover/partial/lease-leader/lease=expiration/run_1/ssh/ssh_145335.046423119_n4_find-ports.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(cluster.go:2344).Run: context canceled
test artifacts and logs in: /artifacts/failover/partial/lease-leader/lease=expiration/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=2
  • ROACHTEST_encrypted=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-23.2 @ 3dec9f224ceb0d881e414d4f7f6238d5a3a3115f:

test decommissionBench/nodes=4/warehouses=1000 failed: (decommissionbench.go:721).runDecommissionBench: monitor failure: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/decommissionBench/nodes=4/warehouses=1000/run_1/ssh/ssh_140446.520926744_n1_cockroach-node-decom.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/decommissionBench/nodes=4/warehouses=1000/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@srosenberg srosenberg mentioned this issue Jul 31, 2024
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-23.2 @ db041dcad77270c616d6e38bf3437f2d1518244e:

test tpccbench/nodes=3/cpu=16 failed: (cluster.go:2230).Start: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpccbench/nodes=3/cpu=16/cpu_arch=arm64/run_1/ssh/ssh_083722.635098831_n2_upload-start-script.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=3/cpu=16/cpu_arch=arm64/run_1

Parameters:

  • ROACHTEST_arch=arm64
  • ROACHTEST_cloud=aws
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-23.2 @ db041dcad77270c616d6e38bf3437f2d1518244e:

test tpcc/multiregion/survive=region/chaos=true failed: (monitor.go:153).Wait: monitor failure: could not stop node :4-6: cluster.StopE: one or more parallel execution failure(s): error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpcc/multiregion/survive=region/chaos=true/run_1/ssh/ssh_200034.358965957_n6_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpcc/multiregion/survive=region/chaos=true/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Aug 2, 2024
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-23.2 @ 29a7ccafb74ff0aae3480a23de77d4a57cfa24b7:

test import/mixed-versions failed: (mixedversion.go:682).Run: mixed-version test failure while running step 2 (start cluster at version "v23.1.2"): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/import/mixed-versions/run_1/ssh/ssh_060654.524670224_n4_upload-start-script.log): TRANSIENT_ERROR(ssh_problem): exit status 255 [owner=test-eng]
test artifacts and logs in: /artifacts/import/mixed-versions/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-23.2 @ 4c8ca4cf727684f1f2acd54da6ff4cf60fdf6587:

test tpccbench/nodes=3/cpu=4/mt-shared-process failed: (cluster.go:2230).Start: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpccbench/nodes=3/cpu=4/mt-shared-process/run_1/ssh/ssh_073255.061382069_n1_upload-start-script.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=3/cpu=4/mt-shared-process/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Aug 14, 2024
Closed
@cockroach-teamcity cockroach-teamcity mentioned this issue Aug 28, 2024
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-23.2 @ 021942d1a7a71bd30fe3aefff6f6a4470cee2a0e:

test follower-reads/survival=zone/locality=regional/reads=exact-staleness failed: (cluster.go:2210).Start: failed to find services to register: failed to find 1 open ports: TRANSIENT_ERROR(open_ports): output:: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/follower-reads/survival=zone/locality=regional/reads=exact-staleness/run_1/ssh/ssh_083606.871853130_n6_find-ports.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/follower-reads/survival=zone/locality=regional/reads=exact-staleness/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

This was referenced Oct 4, 2024
Open
Open
@cockroach-teamcity cockroach-teamcity mentioned this issue Oct 19, 2024
Open
This was referenced Oct 31, 2024
Open
Open
Open
@cockroach-teamcity cockroach-teamcity mentioned this issue Nov 7, 2024
Open
@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-23.2 @ 0391542a84ccbe978ac91bed51faa747e1d68dc5:

test tpch_concurrency/no_streamer failed: (cluster.go:2205).Start: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpch_concurrency/no_streamer/run_1/ssh/ssh_124728.096722732_n3_run-start-script.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpch_concurrency/no_streamer/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-23.2 @ 1a524cd3ced15426926fb4898162ab43c7484d92:

test ycsb/B/nodes=3/cpu=32 failed: (test_runner.go:752).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-arm64" failed: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa -i /home/roach/.ssh/google_compute_engine /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-arm64 [email protected]:./cockroach: signal: killed [owner=test-eng]

Parameters:

  • ROACHTEST_arch=arm64
  • ROACHTEST_cloud=aws
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=32
  • ROACHTEST_encrypted=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

This test on roachdash | Improve this report!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
branch-release-23.2 Used to mark GA and release blockers, technical advisories, and bugs for 23.2 O-roachtest O-robot Originated from a bot. T-testeng TestEng Team X-infra-flake the automatically generated issue was closed due to an infrastructure problem not a product issue
Projects
No open projects
Test Engineering
Status: Triage
Milestone
23.2
Development

No branches or pull requests
1 participant
@cockroach-teamcity