Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

errors from Clickhouse test setup failure could be more explicit #961

Merged
merged 1 commit into from
Apr 22, 2022

Conversation

davepacheco
Copy link
Collaborator

@plotnick saw a test failure that looked similar to this:

$ cargo test -p omicron-nexus --test=test_all -- integration_tests::unauthorized
    Finished test [unoptimized + debuginfo] target(s) in 0.57s
     Running tests/test_all.rs (target/debug/deps/test_all-cfbe7e181ced3561)

running 2 tests
test integration_tests::unauthorized_coverage::test_unauthorized_coverage ... ok
test integration_tests::unauthorized::test_unauthorized ... FAILED

failures:

---- integration_tests::unauthorized::test_unauthorized stdout ----
log file: "/var/folders/x3/6tbrhnss6zz677z4zjsyvkqc0000gn/T/test_all-cfbe7e181ced3561-test_unauthorized.52946.0.log"
note: configured to log to "/var/folders/x3/6tbrhnss6zz677z4zjsyvkqc0000gn/T/test_all-cfbe7e181ced3561-test_unauthorized.52946.0.log"
thread 'integration_tests::unauthorized::test_unauthorized' panicked at 'called `Result::unwrap()` on an `Err` value: No such file or directory (os error 2)', /Users/dap/oxide/omicron/nexus/test-utils/src/lib.rs:99:72
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
WARN: dropped CockroachInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
WARN: temporary directory leaked: /var/folders/x3/6tbrhnss6zz677z4zjsyvkqc0000gn/T/.tmplhhJGt


failures:
    integration_tests::unauthorized::test_unauthorized

test result: FAILED. 1 passed; 1 failed; 0 ignored; 0 measured; 67 filtered out; finished in 0.81s

error: test failed, to rerun pass '-p omicron-nexus --test test_all'

I'm not sure what the failure mode in his case was, but I reproduced it by just not having clickhouse on my PATH. This change adds context to a bunch of error paths in the function that was failing. Now for my failure mode I see this:

$ cargo test -p omicron-nexus --test=test_all -- integration_tests::unauthorized
    Blocking waiting for file lock on build directory
   Compiling omicron-test-utils v0.1.0 (/Users/dap/oxide/omicron/test-utils)
   Compiling nexus-test-utils v0.1.0 (/Users/dap/oxide/omicron/nexus/test-utils)
   Compiling omicron-nexus v0.1.0 (/Users/dap/oxide/omicron/nexus)
    Finished test [unoptimized + debuginfo] target(s) in 1m 40s
     Running tests/test_all.rs (target/debug/deps/test_all-cfbe7e181ced3561)

running 2 tests
test integration_tests::unauthorized_coverage::test_unauthorized_coverage ... ok
test integration_tests::unauthorized::test_unauthorized ... FAILED

failures:

---- integration_tests::unauthorized::test_unauthorized stdout ----
log file: "/var/folders/x3/6tbrhnss6zz677z4zjsyvkqc0000gn/T/test_all-cfbe7e181ced3561-test_unauthorized.53138.0.log"
note: configured to log to "/var/folders/x3/6tbrhnss6zz677z4zjsyvkqc0000gn/T/test_all-cfbe7e181ced3561-test_unauthorized.53138.0.log"
thread 'integration_tests::unauthorized::test_unauthorized' panicked at 'called `Result::unwrap()` on an `Err` value: failed to spawn `clickhouse` (with args: ["server", "--log-file", "/var/folders/x3/6tbrhnss6zz677z4zjsyvkqc0000gn/T/.tmpwfZdb1/clickhouse-server.log", "--errorlog-file", "/var/folders/x3/6tbrhnss6zz677z4zjsyvkqc0000gn/T/.tmpwfZdb1/clickhouse-server.errlog", "--", "--http_port", "0", "--path", "/var/folders/x3/6tbrhnss6zz677z4zjsyvkqc0000gn/T/.tmpwfZdb1"])

Caused by:
    No such file or directory (os error 2)', /Users/dap/oxide/omicron/nexus/test-utils/src/lib.rs:99:72
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
WARN: dropped CockroachInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
WARN: temporary directory leaked: /var/folders/x3/6tbrhnss6zz677z4zjsyvkqc0000gn/T/.tmp78KfQx


failures:
    integration_tests::unauthorized::test_unauthorized

test result: FAILED. 1 passed; 1 failed; 0 ignored; 0 measured; 67 filtered out; finished in 1.09s

error: test failed, to rerun pass '-p omicron-nexus --test test_all'

so instead of just saying "No such file or directory", it says that it got this error while spawning clickhouse.

@davepacheco davepacheco requested a review from bnaecker April 22, 2022 19:17
@davepacheco davepacheco enabled auto-merge (squash) April 22, 2022 19:23
@davepacheco
Copy link
Collaborator Author

@plotnick confirmed the spawn failure is the case he saw, so this should indeed cover that.

@davepacheco davepacheco merged commit 0592c5f into main Apr 22, 2022
@davepacheco davepacheco deleted the clickhouse-test-error branch April 22, 2022 21:05
leftwo pushed a commit that referenced this pull request Oct 4, 2023
Crucible updates
    all Crucible connections should set TCP_NODELAY (#983)
    Use a fixed size for tag and nonce (#957)
    Log crucible opts on start, order crutest options (#974)
    Lock the Downstairs less (#966)
    Cache dirty flag locally, reducing SQLite operations (#970)
    Make stats mutex synchronous (#961)
    Optimize requeue during flow control conditions (#962)
    Update Rust crate base64 to 0.21.4 (#950)
    Do less in control (#949)
    Fix --flush-per-blocks (#959)
    Fast dependency checking (#916)
    Update actions/checkout action to v4 (#960)
    Use `cargo hakari` for better workspace deps (#956)
    Update actions/checkout digest to 8ade135 (#939)
    Cache block size in Guest (#947)
    Update Rust crate ringbuffer to 0.15.0 (#954)
    Update Rust crate toml to 0.8 (#955)
    Update Rust crate reedline to 0.24.0 (#953)
    Update Rust crate libc to 0.2.148 (#952)
    Update Rust crate indicatif to 0.17.7 (#951)
    Remove unused async (#943)
    Use a synchronous mutex for bw/iop_tokens (#946)
    Make flush ID non-locking (#945)
    Use `oneshot` channels instead of `mpsc` for notification (#918)
    Use a strong type for upstairs negotiation (#941)
    Add a "dynamometer" option to crucible-downstairs (#931)
    Get new work and active count in one lock (#938)
    A bunch of misc test cleanup stuff (#937)
    Wait for a snapshot to finish on all downstairs (#920)
    dsc and clippy cleanup. (#935)
    No need to sort ackable_work (#934)
    Use a strong type for repair ID (#928)
    Keep new jobs sorted (#929)
    Remove state_count function on Downstairs (#927)
    Small cleanup to IOStateCount (#932)
    let cmon and IOStateCount use ClientId (#930)
    Fast return for zero length IOs (#926)
    Use a strong type for client ID (#925)
    A few Crucible Agent fixes (#922)
    Use a newtype for `JobId` (#919)
    Don't pass MutexGuard into functions (#917)
    Crutest updates, rename tests, new options (#911)

Propolis updates
    Update tungstenite crates to 0.20
    Use `strum` crate for enum-related utilities
    Wire up bits for CPUID customization
    PHD: improve artifact store (#529)
    Revert abort-on-panic in 'dev' cargo profile
leftwo added a commit that referenced this pull request Oct 5, 2023
Crucible updates
    all Crucible connections should set TCP_NODELAY (#983)
    Use a fixed size for tag and nonce (#957)
    Log crucible opts on start, order crutest options (#974)
    Lock the Downstairs less (#966)
    Cache dirty flag locally, reducing SQLite operations (#970)
    Make stats mutex synchronous (#961)
    Optimize requeue during flow control conditions (#962)
    Update Rust crate base64 to 0.21.4 (#950)
    Do less in control (#949)
    Fix --flush-per-blocks (#959)
    Fast dependency checking (#916)
    Update actions/checkout action to v4 (#960)
    Use `cargo hakari` for better workspace deps (#956)
    Update actions/checkout digest to 8ade135 (#939)
    Cache block size in Guest (#947)
    Update Rust crate ringbuffer to 0.15.0 (#954)
    Update Rust crate toml to 0.8 (#955)
    Update Rust crate reedline to 0.24.0 (#953)
    Update Rust crate libc to 0.2.148 (#952)
    Update Rust crate indicatif to 0.17.7 (#951)
    Remove unused async (#943)
    Use a synchronous mutex for bw/iop_tokens (#946)
    Make flush ID non-locking (#945)
    Use `oneshot` channels instead of `mpsc` for notification (#918)
    Use a strong type for upstairs negotiation (#941)
    Add a "dynamometer" option to crucible-downstairs (#931)
    Get new work and active count in one lock (#938)
    A bunch of misc test cleanup stuff (#937)
    Wait for a snapshot to finish on all downstairs (#920)
    dsc and clippy cleanup. (#935)
    No need to sort ackable_work (#934)
    Use a strong type for repair ID (#928)
    Keep new jobs sorted (#929)
    Remove state_count function on Downstairs (#927)
    Small cleanup to IOStateCount (#932)
    let cmon and IOStateCount use ClientId (#930)
    Fast return for zero length IOs (#926)
    Use a strong type for client ID (#925)
    A few Crucible Agent fixes (#922)
    Use a newtype for `JobId` (#919)
    Don't pass MutexGuard into functions (#917)
    Crutest updates, rename tests, new options (#911)

Propolis updates
    Update tungstenite crates to 0.20
    Use `strum` crate for enum-related utilities
    Wire up bits for CPUID customization
    PHD: improve artifact store (#529)
    Revert abort-on-panic in 'dev' cargo profile

---------

Co-authored-by: Alan Hanson <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants