Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cockroach version check should include stdout/stderr and signal information on failure #1279

Closed
davepacheco opened this issue Jun 24, 2022 · 0 comments

Comments

@davepacheco
Copy link
Collaborator

While trying to reproduce #1130 yet again, I ran into this failure:

ATTEMPT 12177

running 1 test
test integration_tests::disks::test_disk_create_disk_that_already_exists_fails ... FAILED

failures:

---- integration_tests::disks::test_disk_create_disk_that_already_exists_fails stdout ----
log file: "/dangerzone/omicron_tmp/try_repro.5479/test_all-d586ea57740e3382-test_disk_create_disk_that_already_exists_fails.25655.0.log"
note: configured to log to "/dangerzone/omicron_tmp/try_repro.5479/test_all-d586ea57740e3382-test_disk_create_disk_that_already_exists_fails.25655.0.log"
thread 'integration_tests::disks::test_disk_create_disk_that_already_exists_fails' panicked at 'called `Result::unwrap()` on an `Err` value: BadVersion { expected: "v21.2.9", found: Err(error Some(2) when checking CockroachDB version) }', /home/dap/omicron/test-utils/src/dev/mod.rs:141:42
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

The code that produces that message is here:
https://github.com/oxidecomputer/omicron/blob/main/test-utils/src/dev/db.rs#L565-L579

From the code, I think this means that we successfully forked and exec'd cockroach and the process exited normally (in the Unix sense) with status 2 (which is obviously not normal for what we're doing). I found at least one other issue where cockroach did this and it looks like Go must have exited with status 2 after catching SIGILL. Unfortunately, the code that runs this check eats both stdout and stderr. A related issue is that if this process had been terminated by a signal, we wouldn't know which one.

So this issue covers updating this code to record the stdout, stderr, and any signal that may have terminated the process.

leftwo pushed a commit that referenced this issue Apr 30, 2024
Propolis:
Update oximeter dependency to pull in automatic producer registration (#689)
Propagate ReplaceResult up; return disk status (#687)
Enable clippy warnings for lossless casts
Update rustls deps for CVE-2024-32650
migration: refrain from offering all pages when possible (#682)

Crucible:
DTrace probes for IO on/off the network (#1284)
Update oximeter dep to pull in automatic producer registration (#1279)
Remove `ReadResponse` in favor of `RawReadResponse` (#1212)
Fix typo in DTrace upstairs_info (#1276)
replace needing no work should not be an error (#1275)
Add some DTrace scripts to the package. (#1274)
More Pantry updates for Region replacement (#1269)
Send the correct task count for reconciliations (#1271)
Raw extent cleanup (#1268)
leftwo added a commit that referenced this issue Apr 30, 2024
Propolis:
Update oximeter dependency to pull in automatic producer registration (#689)
Propagate ReplaceResult up; return disk status (#687)
Enable clippy warnings for lossless casts
Update rustls deps for CVE-2024-32650
migration: refrain from offering all pages when possible (#682)

Crucible:
DTrace probes for IO on/off the network (#1284)
Update oximeter dep to pull in automatic producer registration (#1279)
Remove `ReadResponse` in favor of `RawReadResponse` (#1212)
Fix typo in DTrace upstairs_info (#1276)
replace needing no work should not be an error (#1275)
Add some DTrace scripts to the package. (#1274)
More Pantry updates for Region replacement (#1269)
Send the correct task count for reconciliations (#1271)
Raw extent cleanup (#1268)

---------

Co-authored-by: Alan Hanson <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant