roachtest: inconsistency failed

[cockroach-teamcity]

(roachtest).inconsistency failed on release-20.2@150c5918fb6e28e0ea5bfa4e8e94088986fbbf98:

The test failed on branch=release-20.2, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/inconsistency/run_1
	inconsistency.go:107,test_runner.go:755: expected a node to crash

More

Artifacts: /inconsistency
Related:

• roachtest: inconsistency failed #56039 roachtest: inconsistency failed C-test-failure O-roachtest O-robot branch-master release-blocker

• roachtest: inconsistency failed #55931 roachtest: inconsistency failed C-test-failure O-roachtest O-robot branch-release-20.1 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

[nvanbenschoten]

The test attempts to inject an inconsistency with:

06:49:24 cluster.go:2270: [w8] > ./cockroach debug pebble db set {store-dir} hex:016b1202000174786e2d0000000000000000000000000000000000 hex:12040800100018002000280032280a10000000000000000000000000000000001a1066616b65207472616e73616374696f6e2a004a00

This hit the following error:

run_064924.592_n1_cockroach_debug_pebble_db_set: 06:49:24 cluster.go:386: > /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2408362-1604124823-22-n3cpu4:1 -- ./cockroach debug pebble db set {store-dir} hex:016b1202000174786e2d0000000000000000000000000000000000 hex:12040800100018002000280032280a10000000000000000000000000000000001a1066616b65207472616e73616374696f6e2a004a00
DB introspection tools

Usage:
  cockroach debug pebble db [command]

Available Commands:
  check       verify checksums and metadata
  lsm         print LSM structure
  properties  print aggregated sstable properties
  scan        print db records
  space       print filesystem space used

Flags:
  -h, --help      help for db
  -v, --verbose   verbose output

Global Flags:
      --logtostderr Severity[=DEFAULT]   logs at or above this threshold go to stderr (default NONE)
      --no-color                         disable standard error log colorization
      --vmodule moduleSpec               comma-separated list of pattern=N settings for file-filtered logging (significantly hurts performance)

Use "cockroach debug pebble db [command] --help" for more information about a command.
run_064924.592_n1_cockroach_debug_pebble_db_set: 06:49:25 cluster.go:2272: > result: <nil>

Did cockroach debug pebble db change recently?

[nvanbenschoten]

From the look of things in pebble/tool/tool.go, cockroach debug pebble db set doesn't exist and hasn't for some time.

[nvanbenschoten]

Oh, this test was recently unskipped: 8732d42. @itsbilal you mention that cockroach debug pebble db set now exists. Where is the code for that? Any idea why this would be saying otherwise?

[itsbilal]

@nvanbenschoten You're looking at the file that includes the db command. Here's where all the sub-commands of db are instantiated:
https://github.com/cockroachdb/pebble/blob/51fc020cf9c0bd579e3ee74357e9d67518ab3722/tool/db.go#L128

This test is failing because db set doesn't exist on release-20.2. The roachtest would need to dynamically detect the version, and run the rocksdb command on 20.2-and-before. Sorry for indirectly breaking and letting you dive into this!

[nvanbenschoten]

Got it, thanks for the explanation!

[nvanbenschoten]

Fixed by #55899.