add stdout/stderr to cockroach version errors

[davepacheco]

During the test suite, when we run cockroach version --build-tag, if we run into an issue like #1876, we don't have the stdout/stderr from Cockroach anywhere. Now, these get dumped out with the error message. This doesn't fix all cases but it's a notable improvement.

[davepacheco]

I happened to do this on top of #1846, so I'm leaving this "draft" so I don't accidentally merge this. If #1846 becomes controversial, I'll rebase this onto "main" instead and submit it separately.

[davepacheco]

Fixes #1279.

[davepacheco]

Here's example output:

$ time while time cargo test -p omicron-nexus --lib -- db:: ; do :; done
...
warning: skipping duplicate package `header-check` found at `/home/dap/.cargo/git/checkouts/propolis-12517f89d3d9f483/c0776be/crates/bhyve-api/header-check`
    Finished test [unoptimized + debuginfo] target(s) in 0.41s
     Running unittests src/lib.rs (target/debug/deps/omicron_nexus-2954a006aadad259)

running 85 tests
test db::collection_attach::test::test_verify_query ... ok
test db::collection_attach::test::test_attach_missing_resource_fails ... ok
test db::collection_detach::test::test_verify_query ... ok
test db::collection_attach::test::test_attach_while_already_attached ... ok
test db::collection_detach::test::test_detach_once ... ok
test db::collection_detach::test::test_detach_deleted_resource_fails ... ok
test db::collection_detach::test::test_detach_missing_collection_fails ... ok
test db::collection_attach::test::test_attach_once_synchronous ... ok
test db::collection_attach::test::test_attach_with_filters ... ok
test db::collection_detach::test::test_detach_without_update_filter ... ok
test db::collection_detach::test::test_detach_while_already_detached ... ok
test db::collection_detach_many::test::test_verify_query ... ok
test db::collection_attach::test::test_attach_once ... ok
test db::collection_attach::test::test_attach_multiple_times ... ok
test db::collection_insert::test::test_verify_query ... ok
test db::datastore::region::test::test_extent_count ... ok
test db::collection_attach::test::test_attach_beyond_capacity_fails ... ok
test db::collection_attach::test::test_attach_without_update_filter ... ok
test db::collection_attach::test::test_attach_missing_collection_fails ... ok
test db::collection_detach::test::test_detach_missing_resource_fails ... ok
test db::collection_attach::test::test_attach_deleted_resource_fails ... ok
test db::collection_detach_many::test::test_detach_many ... ok
test db::collection_detach_many::test::test_detach_deleted_resource ... ok
test db::collection_detach_many::test::test_detach_filter_collection ... ok
test db::collection_detach_many::test::test_detach_missing_collection_fails ... ok
test db::collection_insert::test::test_collection_not_present ... ok
test db::datastore::test::test_deallocate_external_ip_is_idempotent ... ok
test db::collection_detach_many::test::test_detach_once ... ok
test db::datastore::test::test_queries_do_not_require_full_table_scan ... ok
test db::collection_insert::test::test_collection_present ... ok
test db::datastore::test::test_external_ip_check_constraints ... ok
test db::datastore::test::test_deallocate_external_ip_by_instance_id_is_idempotent ... ok
test db::collection_detach_many::test::test_detach_while_already_detached ... ok
test db::fixed_data::role_builtin::test::test_fixed_role_data ... ok
test db::fixed_data::silo_user::test::test_silo_user_ids_are_valid ... ok
test db::fixed_data::test::test_builtin_fleet_id_is_valid ... ok
test db::fixed_data::user_builtin::test::test_builtin_user_ids_are_valid ... ok
test db::datastore::test::test_project_creation ... ok
test db::collection_detach_many::test::test_detach_missing_resource_succeeds ... ok
test db::collection_detach_many::test::test_detach_once_synchronous ... ok
test db::pagination::test::test_paginated_multicolumn_ascending ... FAILED
test db::datastore::test::test_rack_initialize_is_idempotent ... ok
test db::datastore::test::test_region_allocation_out_of_space_fails ... ok
test db::datastore::test::test_region_allocation ... ok
test db::datastore::test::test_region_allocation_is_idempotent ... ok
test db::datastore::test::test_service_upsert ... ok
test db::datastore::test::test_region_allocation_not_enough_datasets ... ok
test db::datastore::test::test_session_methods ... ok
test db::datastore::test::test_ssh_keys ... ok
test db::explain::test::test_explain_full_table_scan ... ok
test db::datastore::test::test_sled_ipv6_address_allocation ... ok
test db::pagination::test::test_paginated_single_column_descending ... ok
test db::pagination::test::test_paginated_single_column_ascending ... ok
test db::lookup::test::test_lookup ... ok
test db::queries::network_interface::tests::test_first_available_address ... ok
test db::explain::test::test_explain_async ... ok
test db::pagination::test::test_paginated_multicolumn_descending ... ok
test db::datastore::test::test_table_scan ... ok
test db::queries::external_ip::tests::test_ensure_pool_exhaustion_does_not_use_other_pool ... ok
test db::queries::external_ip::tests::test_ephemeral_and_snat_ips_do_not_overlap ... ok
test db::queries::external_ip::tests::test_insert_external_ip_for_service_is_idempoent ... ok
test db::queries::external_ip::tests::test_insert_external_ip_for_service_is_idempotent_even_when_full ... ok
test db::queries::external_ip::tests::test_insert_external_ip_is_idempotent ... ok
test db::queries::external_ip::tests::test_next_external_ip_allocation_and_exhaustion ... ok
test db::queries::network_interface::tests::test_last_address_offset ... ok
test db::queries::external_ip::tests::test_next_external_ip_is_restricted_to_pools ... ok
test db::queries::external_ip::tests::test_next_external_ip_is_restricted_to_projects ... ok
test db::queries::external_ip::tests::test_next_external_ip_for_service ... ok
test db::queries::external_ip::tests::test_next_external_ip_out_of_order_allocation_ok ... ok
test db::queries::external_ip::tests::test_next_external_ip_with_ephemeral_takes_whole_port_range ... ok
test db::queries::network_interface::tests::test_insert_multiple_vpcs_fails ... ok
test db::queries::network_interface::tests::test_insert_multiple_vpc_subnets_succeeds ... ok
test db::queries::network_interface::tests::test_detect_ip_exhaustion ... ok
test db::queries::network_interface::tests::test_insert_request_exact_ip ... ok
test db::queries::network_interface::tests::test_insert_no_instance_fails ... ok
test db::queries::network_interface::tests::test_insert_request_same_ip_fails ... ok
test db::queries::network_interface::tests::test_insert_running_instance_fails ... ok
test db::queries::network_interface::tests::test_insert_sequential_ip_allocation ... ok
test db::queries::network_interface::tests::test_insert_same_vpc_subnet_fails ... ok
test db::queries::network_interface::tests::test_limit_number_of_interfaces_per_instance_query ... ok
test db::queries::network_interface::tests::test_insert_with_duplicate_name_fails ... ok
test db::queries::next_item::tests::test_wrapping_next_item_query ... ok
test db::saga_recovery::test::test_failure_during_saga_can_be_recovered ... ok
test db::saga_recovery::test::test_successful_saga_does_not_replay_during_recovery ... ok
test db::queries::vpc_subnet::test::test_filter_conflicting_vpc_subnet_ranges_query ... ok

failures:

---- db::pagination::test::test_paginated_multicolumn_ascending stdout ----
log file: "/dangerzone/omicron_tmp/omicron_nexus-2954a006aadad259-test_paginated_multicolumn_ascending.28293.44.log"
note: configured to log to "/dangerzone/omicron_tmp/omicron_nexus-2954a006aadad259-test_paginated_multicolumn_ascending.28293.44.log"
thread 'db::pagination::test::test_paginated_multicolumn_ascending' panicked at 'failed to start CockroachDB: wrong version of CockroachDB installed. expected 'v22.1.9', found: 'Err(error Some(2) when checking CockroachDB version)
stdout:


stderr:
runtime: s.allocCount= 139 s.nelems= 170
fatal error: s.allocCount != s.nelems && freeIndex == s.nelems

goroutine 1 [running, locked to thread]:
runtime.throw({0x612f6ce, 0x31})
	/opt/ooce/go-1.17/src/runtime/panic.go:1198 +0x74 fp=0xc0009abe28 sp=0xc0009abdf8 pc=0x127d6b4
runtime.(*mcache).nextFree(0xfffffc7fef310fe8, 0xa)
	/opt/ooce/go-1.17/src/runtime/malloc.go:884 +0x228 fp=0xc0009abe68 sp=0xc0009abe28 pc=0x124e6e8
runtime.mallocgc(0x30, 0x5ec4500, 0x1)
	/opt/ooce/go-1.17/src/runtime/malloc.go:1077 +0x530 fp=0xc0009abee8 sp=0xc0009abe68 pc=0x124ec30
runtime.newobject(0x5ec4500)
	/opt/ooce/go-1.17/src/runtime/malloc.go:1234 +0x38 fp=0xc0009abf18 sp=0xc0009abee8 pc=0x124f1d8
runtime.makemap_small()
	/opt/ooce/go-1.17/src/runtime/map.go:293 +0x2d fp=0xc0009abf38 sp=0xc0009abf18 pc=0x124fdcd
github.com/aws/aws-sdk-go/aws/endpoints.init()
	/home/dap/garbage-compactor/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/vendor/github.com/aws/aws-sdk-go/aws/endpoints/defaults.go:8699 +0xb60f3 fp=0xc0009bf7a8 sp=0xc0009abf38 pc=0x4e98593
runtime.doInit(0xaf5a160)
	/opt/ooce/go-1.17/src/runtime/proc.go:6498 +0x129 fp=0xc0009bf8f8 sp=0xc0009bf7a8 pc=0x128e949
runtime.doInit(0xaf63380)
	/opt/ooce/go-1.17/src/runtime/proc.go:6475 +0x7e fp=0xc0009bfa48 sp=0xc0009bf8f8 pc=0x128e89e
runtime.doInit(0xaf788c0)
	/opt/ooce/go-1.17/src/runtime/proc.go:6475 +0x7e fp=0xc0009bfb98 sp=0xc0009bfa48 pc=0x128e89e
runtime.doInit(0xaf58060)
	/opt/ooce/go-1.17/src/runtime/proc.go:6475 +0x7e fp=0xc0009bfce8 sp=0xc0009bfb98 pc=0x128e89e
runtime.doInit(0xafb0ae0)
	/opt/ooce/go-1.17/src/runtime/proc.go:6475 +0x7e fp=0xc0009bfe38 sp=0xc0009bfce8 pc=0x128e89e
runtime.doInit(0xaf503a0)
	/opt/ooce/go-1.17/src/runtime/proc.go:6475 +0x7e fp=0xc0009bff88 sp=0xc0009bfe38 pc=0x128e89e
runtime.main()
	/opt/ooce/go-1.17/src/runtime/proc.go:238 +0x205 fp=0xc0009bffe0 sp=0xc0009bff88 pc=0x12801c5
runtime.goexit()
	/opt/ooce/go-1.17/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0009bffe8 sp=0xc0009bffe0 pc=0x12b6d41

goroutine 18 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.flushDaemon()
	/home/dap/garbage-compactor/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/pkg/util/log/log_flush.go:75 +0x76
created by github.com/cockroachdb/cockroach/pkg/util/log.init.5
	/home/dap/garbage-compactor/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/pkg/util/log/log_flush.go:41 +0x35

goroutine 19 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.signalFlusher()
	/home/dap/garbage-compactor/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/pkg/util/log/log_flush.go:98 +0x145
created by github.com/cockroachdb/cockroach/pkg/util/log.init.5
	/home/dap/garbage-compactor/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/pkg/util/log/log_flush.go:42 +0x4d

goroutine 50 [syscall]:
os/signal.signal_recv()
	/opt/ooce/go-1.17/src/runtime/sigqueue.go:169 +0xab
os/signal.loop()
	/opt/ooce/go-1.17/src/os/signal/signal_unix.go:24 +0x25
created by os/signal.Notify.func1.1
	/opt/ooce/go-1.17/src/os/signal/signal.go:151 +0x3a

goroutine 51 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/goschedstats.init.0.func1()
	/home/dap/garbage-compactor/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/pkg/util/goschedstats/runnable.go:165 +0x1de
created by github.com/cockroachdb/cockroach/pkg/util/goschedstats.init.0
	/home/dap/garbage-compactor/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/pkg/util/goschedstats/runnable.go:157 +0x35

', /home/dap/omicron-cockroachdb/test-utils/src/dev/mod.rs:143:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    db::pagination::test::test_paginated_multicolumn_ascending

test result: FAILED. 84 passed; 1 failed; 0 ignored; 0 measured; 52 filtered out; finished in 23.59s

error: test failed, to rerun pass `-p omicron-nexus --lib`

real	0m24.161s
user	1m57.022s
sys	0m55.510s

real	6m13.403s
user	20m0.017s
sys	8m39.858s

[jgallagher]

LGTM!