roachtest: jepsen/bank-multitable/strobe-skews failed #51728

cockroach-teamcity · 2020-07-22T08:18:03Z

(roachtest).jepsen/bank-multitable/strobe-skews failed on master@e9a4f83e3eee59510f97db2c6e0df9b57cf6b944:

		  | main.runJepsen.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/jepsen.go:159
		  | main.runJepsen.func3
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/jepsen.go:180
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1373
		Wraps: (2) 2 safe details enclosed
		Wraps: (3) output in run_081639.719_n6_bash
		Wraps: (4) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2107908-1595398673-50-n6cpu4:6 -- bash -e -c "\
		  | cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && \
		  |  ~/lein run test \
		  |    --tarball file://${PWD}/cockroach.tgz \
		  |    --username ${USER} \
		  |    --ssh-private-key ~/.ssh/id_rsa \
		  |    --os ubuntu \
		  |    --time-limit 300 \
		  |    --concurrency 30 \
		  |    --recovery-time 25 \
		  |    --test-count 1 \
		  |    -n 10.128.0.123 -n 10.128.0.184 -n 10.128.0.108 -n 10.128.0.188 -n 10.128.0.187 \
		  |    --test bank-multitable --nemesis strobe-skews \
		  | > invoke.log 2>&1 \
		  | " returned
		  | stderr:
		  | Error: SSH_PROBLEM: exit status 255
		  | (1) SSH_PROBLEM
		  | Wraps: (2) Node 6. Command with error:
		  |   | ```
		  |   | bash -e -c "\
		  |   | cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && \
		  |   |  ~/lein run test \
		  |   |    --tarball file://${PWD}/cockroach.tgz \
		  |   |    --username ${USER} \
		  |   |    --ssh-private-key ~/.ssh/id_rsa \
		  |   |    --os ubuntu \
		  |   |    --time-limit 300 \
		  |   |    --concurrency 30 \
		  |   |    --recovery-time 25 \
		  |   |    --test-count 1 \
		  |   |    -n 10.128.0.123 -n 10.128.0.184 -n 10.128.0.108 -n 10.128.0.188 -n 10.128.0.187 \
		  |   |    --test bank-multitable --nemesis strobe-skews \
		  |   | > invoke.log 2>&1 \
		  |   | "
		  |   | ```
		  | Wraps: (3) exit status 255
		  | Error types: (1) errors.SSH (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (5) exit status 10
		Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errutil.withMessage (4) *main.withCommandDetails (5) *exec.ExitError

More

Artifacts: /jepsen/bank-multitable/strobe-skews

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

The text was updated successfully, but these errors were encountered:

cockroach-teamcity · 2020-07-22T08:32:32Z

(roachtest).jepsen/bank/strobe-skews failed on master@e9a4f83e3eee59510f97db2c6e0df9b57cf6b944:

		  | main.runJepsen.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/jepsen.go:159
		  | main.runJepsen.func3
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/jepsen.go:180
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1373
		Wraps: (2) 2 safe details enclosed
		Wraps: (3) output in run_083107.704_n6_bash
		Wraps: (4) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2107908-1595398673-57-n6cpu4:6 -- bash -e -c "\
		  | cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && \
		  |  ~/lein run test \
		  |    --tarball file://${PWD}/cockroach.tgz \
		  |    --username ${USER} \
		  |    --ssh-private-key ~/.ssh/id_rsa \
		  |    --os ubuntu \
		  |    --time-limit 300 \
		  |    --concurrency 30 \
		  |    --recovery-time 25 \
		  |    --test-count 1 \
		  |    -n 10.128.0.106 -n 10.128.0.9 -n 10.128.0.171 -n 10.128.0.170 -n 10.128.0.221 \
		  |    --test bank --nemesis strobe-skews \
		  | > invoke.log 2>&1 \
		  | " returned
		  | stderr:
		  | Error: SSH_PROBLEM: exit status 255
		  | (1) SSH_PROBLEM
		  | Wraps: (2) Node 6. Command with error:
		  |   | ```
		  |   | bash -e -c "\
		  |   | cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && \
		  |   |  ~/lein run test \
		  |   |    --tarball file://${PWD}/cockroach.tgz \
		  |   |    --username ${USER} \
		  |   |    --ssh-private-key ~/.ssh/id_rsa \
		  |   |    --os ubuntu \
		  |   |    --time-limit 300 \
		  |   |    --concurrency 30 \
		  |   |    --recovery-time 25 \
		  |   |    --test-count 1 \
		  |   |    -n 10.128.0.106 -n 10.128.0.9 -n 10.128.0.171 -n 10.128.0.170 -n 10.128.0.221 \
		  |   |    --test bank --nemesis strobe-skews \
		  |   | > invoke.log 2>&1 \
		  |   | "
		  |   | ```
		  | Wraps: (3) exit status 255
		  | Error types: (1) errors.SSH (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (5) exit status 10
		Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errutil.withMessage (4) *main.withCommandDetails (5) *exec.ExitError

More

Artifacts: /jepsen/bank/strobe-skews

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

knz · 2020-07-22T08:56:11Z

caused by #51739

cockroach-teamcity · 2020-07-23T08:10:43Z

(roachtest).jepsen/bank-multitable/strobe-skews failed on master@b8a50cc4d062293915969cdc83e3ec4d057cede5:

		  | main.runJepsen.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/jepsen.go:159
		  | main.runJepsen.func3
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/jepsen.go:180
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1373
		Wraps: (2) 2 safe details enclosed
		Wraps: (3) output in run_080920.976_n6_bash
		Wraps: (4) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2111252-1595484018-48-n6cpu4:6 -- bash -e -c "\
		  | cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && \
		  |  ~/lein run test \
		  |    --tarball file://${PWD}/cockroach.tgz \
		  |    --username ${USER} \
		  |    --ssh-private-key ~/.ssh/id_rsa \
		  |    --os ubuntu \
		  |    --time-limit 300 \
		  |    --concurrency 30 \
		  |    --recovery-time 25 \
		  |    --test-count 1 \
		  |    -n 10.128.0.61 -n 10.128.0.56 -n 10.128.0.65 -n 10.128.0.63 -n 10.128.0.64 \
		  |    --test bank-multitable --nemesis strobe-skews \
		  | > invoke.log 2>&1 \
		  | " returned
		  | stderr:
		  | Error: SSH_PROBLEM: exit status 255
		  | (1) SSH_PROBLEM
		  | Wraps: (2) Node 6. Command with error:
		  |   | ```
		  |   | bash -e -c "\
		  |   | cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && \
		  |   |  ~/lein run test \
		  |   |    --tarball file://${PWD}/cockroach.tgz \
		  |   |    --username ${USER} \
		  |   |    --ssh-private-key ~/.ssh/id_rsa \
		  |   |    --os ubuntu \
		  |   |    --time-limit 300 \
		  |   |    --concurrency 30 \
		  |   |    --recovery-time 25 \
		  |   |    --test-count 1 \
		  |   |    -n 10.128.0.61 -n 10.128.0.56 -n 10.128.0.65 -n 10.128.0.63 -n 10.128.0.64 \
		  |   |    --test bank-multitable --nemesis strobe-skews \
		  |   | > invoke.log 2>&1 \
		  |   | "
		  |   | ```
		  | Wraps: (3) exit status 255
		  | Error types: (1) errors.SSH (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (5) exit status 10
		Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errutil.withMessage (4) *main.withCommandDetails (5) *exec.ExitError

More

Artifacts: /jepsen/bank-multitable/strobe-skews

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-07-23T08:22:33Z

(roachtest).jepsen/bank/strobe-skews failed on master@b8a50cc4d062293915969cdc83e3ec4d057cede5:

		  | main.runJepsen.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/jepsen.go:159
		  | main.runJepsen.func3
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/jepsen.go:180
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1373
		Wraps: (2) 2 safe details enclosed
		Wraps: (3) output in run_082109.971_n6_bash
		Wraps: (4) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2111252-1595484018-55-n6cpu4:6 -- bash -e -c "\
		  | cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && \
		  |  ~/lein run test \
		  |    --tarball file://${PWD}/cockroach.tgz \
		  |    --username ${USER} \
		  |    --ssh-private-key ~/.ssh/id_rsa \
		  |    --os ubuntu \
		  |    --time-limit 300 \
		  |    --concurrency 30 \
		  |    --recovery-time 25 \
		  |    --test-count 1 \
		  |    -n 10.128.0.123 -n 10.128.0.121 -n 10.128.0.37 -n 10.128.0.122 -n 10.128.0.32 \
		  |    --test bank --nemesis strobe-skews \
		  | > invoke.log 2>&1 \
		  | " returned
		  | stderr:
		  | Error: SSH_PROBLEM: exit status 255
		  | (1) SSH_PROBLEM
		  | Wraps: (2) Node 6. Command with error:
		  |   | ```
		  |   | bash -e -c "\
		  |   | cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && \
		  |   |  ~/lein run test \
		  |   |    --tarball file://${PWD}/cockroach.tgz \
		  |   |    --username ${USER} \
		  |   |    --ssh-private-key ~/.ssh/id_rsa \
		  |   |    --os ubuntu \
		  |   |    --time-limit 300 \
		  |   |    --concurrency 30 \
		  |   |    --recovery-time 25 \
		  |   |    --test-count 1 \
		  |   |    -n 10.128.0.123 -n 10.128.0.121 -n 10.128.0.37 -n 10.128.0.122 -n 10.128.0.32 \
		  |   |    --test bank --nemesis strobe-skews \
		  |   | > invoke.log 2>&1 \
		  |   | "
		  |   | ```
		  | Wraps: (3) exit status 255
		  | Error types: (1) errors.SSH (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (5) exit status 10
		Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errutil.withMessage (4) *main.withCommandDetails (5) *exec.ExitError

More

Artifacts: /jepsen/bank/strobe-skews

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

irfansharif · 2020-07-23T16:00:53Z

Fixed by cockroachdb/jepsen#24.

cockroach-teamcity added this to the 20.2 milestone Jul 22, 2020

cockroach-teamcity assigned andreimatei Jul 22, 2020

knz unassigned andreimatei Jul 22, 2020

irfansharif mentioned this issue Jul 23, 2020

cockroachdb: no-longer rely on auto-init cockroachdb/jepsen#24

Merged

irfansharif closed this as completed Jul 23, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

roachtest: jepsen/bank-multitable/strobe-skews failed #51728

roachtest: jepsen/bank-multitable/strobe-skews failed #51728

cockroach-teamcity commented Jul 22, 2020

cockroach-teamcity commented Jul 22, 2020

knz commented Jul 22, 2020

cockroach-teamcity commented Jul 23, 2020

cockroach-teamcity commented Jul 23, 2020

irfansharif commented Jul 23, 2020

roachtest: jepsen/bank-multitable/strobe-skews failed #51728

roachtest: jepsen/bank-multitable/strobe-skews failed #51728

Comments

cockroach-teamcity commented Jul 22, 2020

cockroach-teamcity commented Jul 22, 2020

knz commented Jul 22, 2020

cockroach-teamcity commented Jul 23, 2020

cockroach-teamcity commented Jul 23, 2020

irfansharif commented Jul 23, 2020