Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: ssh_problem failed #127118

Open
cockroach-teamcity opened this issue Jul 13, 2024 · 156 comments
Open

roachtest: ssh_problem failed #127118

cockroach-teamcity opened this issue Jul 13, 2024 · 156 comments
Labels
branch-release-24.2 Used to mark GA and release blockers, technical advisories, and bugs for 24.2 O-roachtest O-robot Originated from a bot. T-testeng TestEng Team X-infra-flake the automatically generated issue was closed due to an infrastructure problem not a product issue

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Jul 13, 2024 •
edited by cockroach-jira-scripts
Loading

roachtest.ssh_problem failed with artifacts on release-24.2 @ 23edce4c9826ebd4711b71c1029d2fc38a25f974:

test ycsb/F/nodes=3/cpu=32 failed: (cluster.go:2417).Run: full command output in run_102438.015309734_n4_workload-run-ycsb-in.log: error persisted after 2 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/ycsb/F/nodes=3/cpu=32/run_1/ssh/ssh_102438.040300454_n4_workload-run-ycsb-in.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/ycsb/F/nodes=3/cpu=32/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=aws
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=32
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-40283
@cockroach-teamcity cockroach-teamcity added branch-release-24.2 Used to mark GA and release blockers, technical advisories, and bugs for 24.2 O-roachtest O-robot Originated from a bot. T-testeng TestEng Team X-infra-flake the automatically generated issue was closed due to an infrastructure problem not a product issue labels Jul 13, 2024
@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.2 @ d4ef0343f38a35fb781b6343697b57a58131c68e:

test allocbench/nodes=7/cpu=8/kv/r=0/ops=skew failed: (test_runner.go:811).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64" failed: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64 [email protected]:./cockroach
ssh: connect to host 35.196.70.57 port 22: Connection timed out
lost connection: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.2 @ d4ef0343f38a35fb781b6343697b57a58131c68e:

test failover/non-system/disk-stall/read-only/read-write failed: (test_runner.go:811).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64" failed: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64 [email protected]:./cockroach
ssh: connect to host 34.75.165.77 port 22: Connection timed out
lost connection: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=2
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.2 @ 238d10a22cff57ddc8393b149cffb58883a2a89f:

test allocbench/nodes=7/cpu=8/kv/r=0/access=skew failed: (test_runner.go:811).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64" failed: error persisted after 2 attempts: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64 [email protected]:./cockroach
Warning: Permanently added '35.190.176.150' (ECDSA) to the list of known hosts.
client_loop: send disconnect: Broken pipe
lost connection: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.2 @ 238d10a22cff57ddc8393b149cffb58883a2a89f:

test cdc/kafka-topics failed: (cluster.go:2303).Start: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/kafka-topics/run_1/ssh/ssh_094353.220580245_n1_init-cluster.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/cdc/kafka-topics/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Jul 16, 2024
Open
@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.2 @ 238d10a22cff57ddc8393b149cffb58883a2a89f:

test cdc/workload/kv100/nodes=5/cpu=16/ranges=100/server=scheduler/protocol=mux/format=json/sink=null failed: (test_runner.go:811).func4: one or more parallel execution failure(s): error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/_runner-logs/cluster-create/ssh/ssh_131954.172690916_n2_ssh-dist-key.log): TRANSIENT_ERROR(ssh_problem): exit status 255 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.2 @ 238d10a22cff57ddc8393b149cffb58883a2a89f:

test kv0/enc=false/nodes=3/cpu=32/size=64kb failed: (test_runner.go:811).func4: cluster.PutE: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/_runner-logs/ssh/ssh_160039.349556385_n1_if-hostname-teamcity.log): TRANSIENT_ERROR(ssh_problem): exit status 255 [owner=test-eng]

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=32
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

This was referenced Jul 17, 2024
Open
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.2 @ 037402f139a1d037d40cd9783431f3790fa3b75a:

test cdc/scan/initial/nodes=5/cpu=16/rows=1G/ranges=100/protocol=mux/format=json/sink=null failed: (test_runner.go:811).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-arm64" failed: error persisted after 2 attempts: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-arm64 [email protected]:./cockroach
Warning: Permanently added '34.123.189.80' (ECDSA) to the list of known hosts.
client_loop: send disconnect: Broken pipe
lost connection: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_arch=arm64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.2 @ 2698d283dbc1eae3b14acf0912fd61777297716a:

test allocbench/nodes=7/cpu=8/kv/r=0/access=skew failed: (test_runner.go:811).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64" failed: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64 [email protected]:./cockroach
ssh: connect to host 34.75.45.56 port 22: Connection timed out
lost connection: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Jul 18, 2024
Open
@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.2 @ 2698d283dbc1eae3b14acf0912fd61777297716a:

test kv0/enc=false/nodes=3/cpu=32 failed: (test_runner.go:811).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64" failed: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64 [email protected]:./cockroach
ssh: connect to host 104.196.144.160 port 22: Connection timed out
lost connection: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=32
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.2 @ 3a5a06709653819261ee998badf5efd6d3bd5955:

test import/tpcc/warehouses=1000/nodes=32 failed: (test_runner.go:811).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64" failed: error persisted after 2 attempts: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64 [email protected]:./cockroach
Warning: Permanently added '34.139.130.173' (ECDSA) to the list of known hosts.
client_loop: send disconnect: Broken pipe
lost connection: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.2 @ 3a5a06709653819261ee998badf5efd6d3bd5955:

test sysbench/oltp_read_write/nodes=3/cpu=32/conc=256 failed: (test_runner.go:811).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64" failed: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64 [email protected]:./cockroach
ssh: connect to host 35.237.196.144 port 22: Connection timed out
lost connection: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=32
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.2 @ 3a5a06709653819261ee998badf5efd6d3bd5955:

test ycsb/F/nodes=3/cpu=32/isolation-level=read-committed failed: (test_runner.go:811).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64" failed: error persisted after 2 attempts: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64 [email protected]:./cockroach
Warning: Permanently added '34.139.107.164' (ECDSA) to the list of known hosts.
client_loop: send disconnect: Broken pipe
lost connection: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=32
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.2 @ 60d3fb84a73f5e31c99404223f3a0ba91e67f489:

test allocbench/nodes=7/cpu=8/kv/r=95/ops=skew failed: (test_runner.go:811).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-arm64" failed: error persisted after 2 attempts: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-arm64 [email protected]:./cockroach
Warning: Permanently added '34.27.147.249' (ECDSA) to the list of known hosts.
client_loop: send disconnect: Broken pipe
lost connection: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_arch=arm64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.2 @ 60d3fb84a73f5e31c99404223f3a0ba91e67f489:

test cdc/workload/kv100/nodes=5/cpu=16/ranges=100k/server=processor/protocol=mux/format=json/sink=null failed: (test_runner.go:811).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64" failed: error persisted after 2 attempts: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64 [email protected]:./cockroach
client_loop: send disconnect: Broken pipe
lost connection: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.2 @ 60d3fb84a73f5e31c99404223f3a0ba91e67f489:

test import/tpcc/warehouses=1000/nodes=32 failed: (test_runner.go:811).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64" failed: error persisted after 2 attempts: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64 [email protected]:./cockroach
ssh: connect to host 34.139.55.169 port 22: Connection refused
lost connection: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.2 @ 60d3fb84a73f5e31c99404223f3a0ba91e67f489:

test ycsb/D/nodes=3/cpu=32/isolation-level=read-committed failed: (test_runner.go:811).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64" failed: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64 [email protected]:./cockroach
ssh: connect to host 34.73.210.71 port 22: Connection timed out
lost connection: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=32
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.2 @ cf1aaffe79dad34124399658a66ba2379a499047:

test allocbench/nodes=7/cpu=8/kv/r=95/ops=skew failed: (test_runner.go:811).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64" failed: error persisted after 2 attempts: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64 [email protected]:./cockroach
Warning: Permanently added '35.185.28.99' (ECDSA) to the list of known hosts.
client_loop: send disconnect: Broken pipe
lost connection: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.2 @ cf1aaffe79dad34124399658a66ba2379a499047:

test failover/partial/lease-leader failed: (test_runner.go:811).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64" failed: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64 [email protected]:./cockroach
ssh: connect to host 34.75.85.37 port 22: Connection timed out
lost connection: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=2
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.2 @ 509b1e919fa3e89254b4b836854aaddc5c26fc4c:

test follower-reads/mixed-version/survival=region/locality=global/reads=strong failed: (mixedversion.go:645).Run: mixed-version test failure while running step 59 (restart node 5 with binary version v24.1.2): cluster.StopE: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/follower-reads/mixed-version/survival=region/locality=global/reads=strong/run_1/ssh/ssh_150620.962382583_n5_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/follower-reads/mixed-version/survival=region/locality=global/reads=strong/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.2 @ 17ee613b96801d4c43e9fa9d20ebe23f29d9c89e:

test cdc/workload/kv100/nodes=5/cpu=16/ranges=100/server=scheduler/protocol=mux/format=json/sink=null failed: (test_runner.go:811).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64" failed: error persisted after 2 attempts: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64 [email protected]:./cockroach
Warning: Permanently added '34.148.191.250' (ECDSA) to the list of known hosts.
client_loop: send disconnect: Broken pipe
lost connection: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

This was referenced Nov 1, 2024
Open
Open
@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.2 @ 2112e82243d0f454e659236dbf76937ce73dc013:

test failover/non-system/blackhole/lease=expiration failed: (test_runner.go:815).func4: cluster.PutE: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/_runner-logs/ssh/ssh_110342.173801732_n1_if-hostname-teamcity.log): TRANSIENT_ERROR(ssh_problem): exit status 255 [owner=test-eng]

Parameters:

  • ROACHTEST_arch=arm64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=2
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=false
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Nov 7, 2024
Open
@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.2 @ 2112e82243d0f454e659236dbf76937ce73dc013:

test tpch_concurrency failed: (cluster.go:2331).Stop: cluster.StopE: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpch_concurrency/run_1/ssh/ssh_170720.700587914_n3_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpch_concurrency/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.2 @ 2112e82243d0f454e659236dbf76937ce73dc013:

test tpcc/mixed-headroom/n5cpu16 failed: (mixedversion.go:705).Run: mixed-version test failure while running step 61 (run "check TPCC workload"): full command output in run_182152.506536248_n5_cockroach-workload-c.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpcc/mixed-headroom/n5cpu16/run_1/ssh/ssh_182152.518569994_n5_cockroach-workload-c.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpcc/mixed-headroom/n5cpu16/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.2 @ 5764366875adca70c3a3ff3d4d13aaf599bc53a7:

test acceptance/mismatched-locality failed: (test_runner.go:815).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64" failed: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa -i /home/roach/.ssh/google_compute_engine /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64 [email protected]:./cockroach
ssh: connect to host 34.138.174.231 port 22: Connection timed out
lost connection: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.2 @ 5764366875adca70c3a3ff3d4d13aaf599bc53a7:

test loqrecovery/workload=tpcc/rangeSize=16mb failed: (test_runner.go:815).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-arm64" failed: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa -i /home/roach/.ssh/google_compute_engine /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-arm64 [email protected]:./cockroach
ssh: connect to host 34.41.83.8 port 22: Connection timed out
lost connection: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_arch=arm64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=false
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.2 @ be69012d37da70d0ada6eb4cf2b008d02ef16ffd:

test c2c/shutdown/src/coordinator failed: (test_runner.go:815).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-arm64" failed: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa -i /home/roach/.ssh/google_compute_engine /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-arm64 [email protected]:./cockroach
ssh: connect to host 104.198.213.53 port 22: Connection timed out
lost connection: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_arch=arm64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=false
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Nov 12, 2024
Open
@cockroach-teamcity
Copy link
Member Author

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.ssh_problem failed with artifacts on release-24.2 @ c9b7151258520849effaa3abd9514f00fb240039:

test rebalance/by-load/leases/mixed-version failed: (test_runner.go:815).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach-ea.linux-amd64" failed: error persisted after 2 attempts: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa -i /home/roach/.ssh/google_compute_engine /go/src/github.com/cockroachdb/cockroach/bin/cockroach-ea.linux-amd64 [email protected]:./cockroach
Connection to 35.185.55.27 closed by remote host.
lost connection: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_runtimeAssertionsBuild=true
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.2 @ 287e165b88ff5aa1aa3c9b9ff1303d859c1f960e:

test cdc/scan/catchup-cold/nodes=5/cpu=16/rows=1G/ranges=100/protocol=mux/format=json/sink=null failed: (test_runner.go:815).func4: cluster.PutE: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/_runner-logs/ssh/ssh_084559.902575966_n1_if-hostname-teamcity.log): TRANSIENT_ERROR(ssh_problem): exit status 255 [owner=test-eng]

Parameters:

  • ROACHTEST_arch=arm64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=false
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.2 @ ce4c11fbe7ea10e03d44a26ad3f0d483f01c619b:

test acceptance/version-upgrade failed: (mixedversion.go:710).Run: mixed-version test failure while running step 27 (restart node 1 with binary version v24.1.6): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/acceptance/version-upgrade/run_1/ssh/ssh_080034.240155710_n1_upload-start-script.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/acceptance/version-upgrade/run_1

Parameters:

  • arch=amd64
  • cloud=aws
  • coverageBuild=false
  • cpu=4
  • encrypted=false
  • fs=ext4
  • localSSD=true
  • mvtDeploymentMode=shared-process
  • mvtVersions=v23.1.28 → v23.2.14 → v24.1.6 → release-24.2
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.2 @ ce4c11fbe7ea10e03d44a26ad3f0d483f01c619b:

test cdc/workload/kv0/nodes=5/cpu=16/ranges=100k/control failed: (test_runner.go:817).func4: cluster.PutE: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/_runner-logs/ssh/ssh_093457.006702045_n1_if-hostname-teamcity.log): TRANSIENT_ERROR(ssh_problem): exit status 255 [owner=test-eng]

Parameters:

  • arch=arm64
  • cloud=gce
  • coverageBuild=false
  • cpu=16
  • encrypted=false
  • fs=ext4
  • localSSD=false
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.2 @ ce4c11fbe7ea10e03d44a26ad3f0d483f01c619b:

test decommission/randomized failed: (test_runner.go:817).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-arm64" failed: error persisted after 2 attempts: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa -i /home/roach/.ssh/google_compute_engine /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-arm64 [email protected]:./cockroach
Warning: Permanently added '34.132.231.144' (ECDSA) to the list of known hosts.
client_loop: send disconnect: Broken pipe
lost connection: exit status 1 [owner=test-eng]

Parameters:

  • arch=arm64
  • cloud=gce
  • coverageBuild=false
  • cpu=4
  • encrypted=false
  • fs=ext4
  • localSSD=false
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.2 @ c427c814a2b10c42b1eee634031294eb74d15aa6:

test jepsen/bank-multitable/subcritical-skews failed: (test_runner.go:817).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-arm64" failed: error persisted after 2 attempts: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa -i /home/roach/.ssh/google_compute_engine /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-arm64 [email protected]:./cockroach
Warning: Permanently added '104.198.131.15' (ECDSA) to the list of known hosts.
client_loop: send disconnect: Broken pipe
lost connection: exit status 1 [owner=test-eng]

Parameters:

  • arch=arm64
  • cloud=gce
  • coverageBuild=false
  • cpu=4
  • encrypted=false
  • fs=ext4
  • localSSD=false
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.2 @ c427c814a2b10c42b1eee634031294eb74d15aa6:

test alterpk-bank failed: (test_runner.go:817).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64" failed: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa -i /home/roach/.ssh/google_compute_engine /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64 [email protected]:./cockroach
ssh: connect to host 35.231.45.233 port 22: Connection timed out
lost connection: exit status 1 [owner=test-eng]

Parameters:

  • arch=amd64
  • cloud=gce
  • coverageBuild=false
  • cpu=4
  • encrypted=false
  • fs=ext4
  • localSSD=true
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.2 @ c427c814a2b10c42b1eee634031294eb74d15aa6:

test acceptance/gossip/restart-node-one failed: (test_runner.go:817).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-arm64" failed: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa -i /home/roach/.ssh/google_compute_engine /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-arm64 [email protected]:./cockroach
Warning: Permanently added '34.171.112.75' (ECDSA) to the list of known hosts.
[email protected]: Permission denied (publickey).
lost connection: exit status 1 [owner=test-eng]

Parameters:

  • arch=arm64
  • cloud=gce
  • coverageBuild=false
  • cpu=4
  • encrypted=false
  • fs=ext4
  • localSSD=false
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.2 @ c427c814a2b10c42b1eee634031294eb74d15aa6:

test failover/chaos/read-write failed: (test_runner.go:817).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-arm64" failed: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa -i /home/roach/.ssh/google_compute_engine /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-arm64 [email protected]:./cockroach
ssh: connect to host 34.136.185.116 port 22: Connection timed out
lost connection: exit status 1 [owner=test-eng]

Parameters:

  • arch=arm64
  • cloud=gce
  • coverageBuild=false
  • cpu=2
  • encrypted=false
  • fs=ext4
  • localSSD=false
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.2 @ c427c814a2b10c42b1eee634031294eb74d15aa6:

test jepsen/sets/start-stop-2 failed: (test_runner.go:817).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64" failed: error persisted after 2 attempts: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa -i /home/roach/.ssh/google_compute_engine /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64 [email protected]:./cockroach
client_loop: send disconnect: Broken pipe
lost connection: exit status 1 [owner=test-eng]

Parameters:

  • arch=amd64
  • cloud=gce
  • coverageBuild=false
  • cpu=4
  • encrypted=false
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.2 @ 1cb22e6a88b7023baa19934c627f2aace244f5d7:

test c2c/kv0 failed: (test_runner.go:817).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-arm64" failed: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa -i /home/roach/.ssh/google_compute_engine /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-arm64 [email protected]:./cockroach
ssh: connect to host 34.45.238.218 port 22: Connection timed out
lost connection: exit status 1 [owner=test-eng]

Parameters:

  • arch=arm64
  • cloud=gce
  • coverageBuild=false
  • cpu=8
  • encrypted=false
  • fs=ext4
  • localSSD=false
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.2 @ c05cf9e2a2bbc0db07fd7758c3cfa3bff4ed22ec:

test allocbench/nodes=7/cpu=8/kv/r=0/access=skew failed: (test_runner.go:817).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64" failed: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa -i /home/roach/.ssh/google_compute_engine /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64 [email protected]:./cockroach
ssh: connect to host 34.138.141.84 port 22: Connection timed out
lost connection: exit status 1 [owner=test-eng]

Parameters:

  • arch=amd64
  • cloud=gce
  • coverageBuild=false
  • cpu=8
  • encrypted=false
  • fs=ext4
  • localSSD=true
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.2 @ ef9ea88d4da8870e45b8a37ff7b0b504b6d77135:

test failover/chaos/read-write failed: (test_runner.go:823).func4: cluster.PutE: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/_runner-logs/ssh/ssh_105246.593422906_n1_if-hostname-teamcity.log): TRANSIENT_ERROR(ssh_problem): exit status 255 [owner=test-eng]

Parameters:

  • arch=arm64
  • cloud=gce
  • coverageBuild=false
  • cpu=2
  • encrypted=false
  • fs=ext4
  • localSSD=false
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.ssh_problem failed with artifacts on release-24.2 @ 1ba69a2843ef6f3ffdd8a3863de571342aa501eb:

test alterpk-tpcc-500 failed: (test_runner.go:823).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach-ea.linux-amd64" failed: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa -i /home/roach/.ssh/google_compute_engine /go/src/github.com/cockroachdb/cockroach/bin/cockroach-ea.linux-amd64 [email protected]:./cockroach
ssh: connect to host 34.148.229.109 port 22: Connection timed out
lost connection: exit status 1 [owner=test-eng]

Parameters:

  • arch=amd64
  • cloud=gce
  • coverageBuild=false
  • cpu=16
  • encrypted=false
  • fs=ext4
  • localSSD=true
  • runtimeAssertionsBuild=true
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.2 @ 6494c4e0e13d6f805e8e58396279dae1fd2e6fd9:

test acceptance/version-upgrade failed: (test_runner.go:823).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64" failed: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa -i /home/roach/.ssh/google_compute_engine /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64 [email protected]:./cockroach
ssh: connect to host 34.74.233.6 port 22: Connection timed out
lost connection: exit status 1 [owner=test-eng]

Parameters:

  • arch=amd64
  • cloud=gce
  • coverageBuild=false
  • cpu=4
  • encrypted=false
  • fs=ext4
  • localSSD=true
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.2 @ 6494c4e0e13d6f805e8e58396279dae1fd2e6fd9:

test allocbench/nodes=7/cpu=8/kv/r=0/ops=skew failed: (test_runner.go:823).func4: cluster.PutE: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/_runner-logs/ssh/ssh_065409.690823404_n1_if-hostname-teamcity.log): TRANSIENT_ERROR(ssh_problem): exit status 255 [owner=test-eng]

Parameters:

  • arch=amd64
  • cloud=gce
  • coverageBuild=false
  • cpu=8
  • encrypted=false
  • fs=ext4
  • localSSD=true
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.2 @ 6494c4e0e13d6f805e8e58396279dae1fd2e6fd9:

test allocbench/nodes=7/cpu=8/kv/r=0/access=skew failed: (test_runner.go:823).func4: cluster.PutE: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/_runner-logs/ssh/ssh_065516.869168269_n1_if-hostname-teamcity.log): TRANSIENT_ERROR(ssh_problem): exit status 255 [owner=test-eng]

Parameters:

  • arch=amd64
  • cloud=gce
  • coverageBuild=false
  • cpu=8
  • encrypted=false
  • fs=ext4
  • localSSD=true
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.2 @ 9dab6cb9f71f21ef1e9c98640193035783ba9a4e:

test acceptance/decommission-self failed: (test_runner.go:823).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-arm64" failed: error persisted after 2 attempts: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa -i /home/roach/.ssh/google_compute_engine /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-arm64 [email protected]:./cockroach
Warning: Permanently added '34.121.217.43' (ECDSA) to the list of known hosts.
client_loop: send disconnect: Broken pipe
lost connection: exit status 1 [owner=test-eng]

Parameters:

  • arch=arm64
  • cloud=gce
  • coverageBuild=false
  • cpu=4
  • encrypted=false
  • fs=ext4
  • localSSD=false
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Nov 26, 2024
Open
@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.2 @ 9dab6cb9f71f21ef1e9c98640193035783ba9a4e:

test c2c/shutdown/src/coordinator failed: VMs preempted during the test run: teamcity-17918622-1732603274-38-n9cpu8-0006 (us-central1-f), teamcity-17918622-1732603274-38-n9cpu8-0007 (us-central1-f), teamcity-17918622-1732603274-38-n9cpu8-0009 (us-central1-f)

**Other Failures:**
(cluster.go:2398).Run: full command output in run_085226.910947650_n9_cockroach-workload-i.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/c2c/shutdown/src/coordinator/cpu_arch=arm64/run_1/ssh/ssh_085226.960558983_n9_cockroach-workload-i.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(test_runner.go:1294).runTest: test timed out (20m0s)
test artifacts and logs in: /artifacts/c2c/shutdown/src/coordinator/cpu_arch=arm64/run_1

Parameters:

  • arch=arm64
  • cloud=gce
  • coverageBuild=false
  • cpu=8
  • encrypted=false
  • fs=ext4
  • localSSD=false
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
branch-release-24.2 Used to mark GA and release blockers, technical advisories, and bugs for 24.2 O-roachtest O-robot Originated from a bot. T-testeng TestEng Team X-infra-flake the automatically generated issue was closed due to an infrastructure problem not a product issue
Projects
No open projects
Test Engineering
Status: Triage
Milestone
No milestone
Development

No branches or pull requests
1 participant
@cockroach-teamcity