roachtest: kv/splits/nodes=3/quiesce=true failed #93579

cockroach-teamcity · 2022-12-14T07:13:07Z

roachtest.kv/splits/nodes=3/quiesce=true failed with artifacts on master @ 052acc88ad9d7296ce6b8b441627fb469cc74d95:

test artifacts and logs in: /artifacts/kv/splits/nodes=3/quiesce=true/run_1
(test_impl.go:291).Fatal: output in run_062628.605496985_n4_workload_run_kv: ./workload run kv --init --max-ops=1 --concurrency=192 --splits=300000 {pgurl:1-3} returned: context canceled
(test_impl.go:291).Fatal: monitor failure: monitor command failure: unexpected node event: 1: dead (exit status 137)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/kv-triage _{This test on roachdash | Improve this report!

Jira issue: CRDB-22415
Epic CRDB-18656}

The text was updated successfully, but these errors were encountered:

kvoli · 2022-12-14T07:26:59Z

Node 1 OOM'd here. This is most likely due to #92858.

The additional memory overhead per replica introduced in that patch caused the node to oom before it could reach 300k replicas.

I'll look into a fix tomorrow, this wasn't unexpected but it motivates pursuing Stat consolidation more seriously now #87187.

wip resolves cockroachdb#87187 resolves cockroachdb#93579 Release note: None

pav-kv · 2022-12-14T16:47:02Z

@kvoli How significant is this memory overhead? Was this test already close to the OOM threshold?

I.e. should we leave the release-blocker label?

kvoli · 2022-12-14T16:49:44Z

It was already close to the threshold. This added approx 2kb per replica, So 600 mb on this test.

pav-kv · 2022-12-14T17:35:45Z

Sounds non-trivial to me, although I don't know a typical Replica footprint. Do you know what percentage that is?

cockroach-teamcity · 2022-12-15T07:09:11Z

roachtest.kv/splits/nodes=3/quiesce=true failed with artifacts on master @ d6f98e90684894fd36f53596e6aac355676d232e:

test artifacts and logs in: /artifacts/kv/splits/nodes=3/quiesce=true/run_1
(test_impl.go:291).Fatal: output in run_062230.511601923_n4_workload_run_kv: ./workload run kv --init --max-ops=1 --concurrency=192 --splits=300000 {pgurl:1-3} returned: context canceled
(test_impl.go:291).Fatal: monitor failure: monitor command failure: unexpected node event: 1: dead (exit status 137)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

kvoli · 2022-12-15T16:36:45Z

The test fails at 298k ranges (/300k).

Sounds non-trivial to me, although I don't know a typical Replica footprint. Do you know what percentage that is?

In the most recent failed test, the heap pprof for the oom node showed 33% of total heap mem was used for replica stats (kv only I believe)

This is from 72% total associated with replicas:

It seems to me like about 15kb per replica from the heap pprof and 8kb is attributable to replicastats.

I'm going to work on a patch now to reduce this by a constant factor per replica. Longer term I think this needs to have a lower variable cost < 1kb and a fixed cost per store.

cockroach-teamcity · 2022-12-18T07:12:00Z

roachtest.kv/splits/nodes=3/quiesce=true failed with artifacts on master @ 93ed65565357538c9048ff45c878a493f2ed9b45:

test artifacts and logs in: /artifacts/kv/splits/nodes=3/quiesce=true/run_1
(test_impl.go:291).Fatal: output in run_062255.852512248_n4_workload_run_kv: ./workload run kv --init --max-ops=1 --concurrency=192 --splits=300000 {pgurl:1-3} returned: context canceled
(test_impl.go:291).Fatal: monitor failure: monitor command failure: unexpected node event: 3: dead (exit status 137)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

kvoli · 2022-12-19T14:40:24Z

I have a patch that is a medium term fix for this #93823. It reduces the memory footprint per-replica of load stats by 50% (4gb avail space now).

Longer term it would be ideal to maintain only counters on the replica and then have a fixed number of replicas tracked using more memory intensive moving average strategies.

cockroach-teamcity · 2022-12-22T07:08:22Z

roachtest.kv/splits/nodes=3/quiesce=true failed with artifacts on master @ 15bc0c4780681d95e705c22dbe2e7d8d190054ef:

test artifacts and logs in: /artifacts/kv/splits/nodes=3/quiesce=true/run_1
(test_impl.go:291).Fatal: output in run_062300.618391764_n4_workload_run_kv: ./workload run kv --init --max-ops=1 --concurrency=192 --splits=300000 {pgurl:1-3} returned: context canceled
(test_impl.go:291).Fatal: monitor failure: monitor command failure: unexpected node event: 1: dead (exit status 137)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-12-23T07:08:45Z

roachtest.kv/splits/nodes=3/quiesce=true failed with artifacts on master @ 4232883add85a151c423c45904ac4096d04656c5:

test artifacts and logs in: /artifacts/kv/splits/nodes=3/quiesce=true/run_1
(test_impl.go:291).Fatal: output in run_062210.708400693_n4_workload_run_kv: ./workload run kv --init --max-ops=1 --concurrency=192 --splits=300000 {pgurl:1-3} returned: context canceled
(test_impl.go:291).Fatal: monitor failure: monitor command failure: unexpected node event: 3: dead (exit status 137)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-12-25T07:09:19Z

roachtest.kv/splits/nodes=3/quiesce=true failed with artifacts on master @ c4bde8b72cdd4016845ae70ef5162b3f11fab1fb:

test artifacts and logs in: /artifacts/kv/splits/nodes=3/quiesce=true/run_1
(test_impl.go:291).Fatal: output in run_062239.427583472_n4_workload_run_kv: ./workload run kv --init --max-ops=1 --concurrency=192 --splits=300000 {pgurl:1-3} returned: context canceled
(test_impl.go:291).Fatal: monitor failure: monitor command failure: unexpected node event: 3: dead (exit status 137)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-12-26T07:07:33Z

roachtest.kv/splits/nodes=3/quiesce=true failed with artifacts on master @ c4bde8b72cdd4016845ae70ef5162b3f11fab1fb:

test artifacts and logs in: /artifacts/kv/splits/nodes=3/quiesce=true/run_1
(test_impl.go:291).Fatal: output in run_062255.354302199_n4_workload_run_kv: ./workload run kv --init --max-ops=1 --concurrency=192 --splits=300000 {pgurl:1-3} returned: context canceled
(test_impl.go:291).Fatal: monitor failure: monitor command failure: unexpected node event: 1: dead (exit status 137)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-12-27T07:11:19Z

roachtest.kv/splits/nodes=3/quiesce=true failed with artifacts on master @ 9c5375f6a7375724cdbcbaa0029ed97a230d7abe:

test artifacts and logs in: /artifacts/kv/splits/nodes=3/quiesce=true/run_1
(test_impl.go:291).Fatal: output in run_062722.926130212_n4_workload_run_kv: ./workload run kv --init --max-ops=1 --concurrency=192 --splits=300000 {pgurl:1-3} returned: context canceled
(test_impl.go:291).Fatal: monitor failure: monitor command failure: unexpected node event: 1: dead (exit status 137)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

kvoli · 2022-12-28T18:54:12Z

#93823 should resolve this failing test. It would be nice to enforce a lower per-replica memory budget by increasing the number of splits. This test failing would catch creeping heap allocations earlier.

We could increase the splits in this test (or add another test) to 375-400k w/ the 14gb of available memory (4gb headroom on that patch).

cockroach-teamcity · 2022-12-29T07:09:23Z

roachtest.kv/splits/nodes=3/quiesce=true failed with artifacts on master @ 64e4fc9faa4e0ab19fe5ba78f053bc2b1390cb5e:

test artifacts and logs in: /artifacts/kv/splits/nodes=3/quiesce=true/run_1
(cluster.go:1933).Run: output in run_062310.145278928_n4_workload_run_kv: ./workload run kv --init --max-ops=1 --concurrency=192 --splits=300000 {pgurl:1-3} returned: context canceled
(monitor.go:127).Wait: monitor failure: monitor command failure: unexpected node event: 1: dead (exit status 137)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-12-30T07:18:48Z

roachtest.kv/splits/nodes=3/quiesce=true failed with artifacts on master @ 250238cd29102391dddbc8cc71380090c49ce509:

test artifacts and logs in: /artifacts/kv/splits/nodes=3/quiesce=true/run_1
(cluster.go:1933).Run: output in run_062649.065426369_n4_workload_run_kv: ./workload run kv --init --max-ops=1 --concurrency=192 --splits=300000 {pgurl:1-3} returned: context canceled
(monitor.go:127).Wait: monitor failure: monitor command failure: unexpected node event: 1: dead (exit status 137)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-01-02T07:09:13Z

roachtest.kv/splits/nodes=3/quiesce=true failed with artifacts on master @ 0725273ac7f789ba8ed78aacaf73cc953ca47fe8:

test artifacts and logs in: /artifacts/kv/splits/nodes=3/quiesce=true/run_1
(cluster.go:1933).Run: output in run_062201.133474248_n4_workload_run_kv: ./workload run kv --init --max-ops=1 --concurrency=192 --splits=300000 {pgurl:1-3} returned: context canceled
(monitor.go:127).Wait: monitor failure: monitor command failure: unexpected node event: 2: dead (exit status 137)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-01-03T07:07:33Z

roachtest.kv/splits/nodes=3/quiesce=true failed with artifacts on master @ 1d7bd69205c2197ccac33df9e2e6d4ff8c0fdbcf:

test artifacts and logs in: /artifacts/kv/splits/nodes=3/quiesce=true/run_1
(cluster.go:1933).Run: output in run_062226.772540067_n4_workload_run_kv: ./workload run kv --init --max-ops=1 --concurrency=192 --splits=300000 {pgurl:1-3} returned: context canceled
(monitor.go:127).Wait: monitor failure: monitor command failure: unexpected node event: 2: dead (exit status 137)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-01-04T06:59:51Z

roachtest.kv/splits/nodes=3/quiesce=true failed with artifacts on master @ 2c3d75f1ce31024d7ffe530f91f22162c053abcd:

test artifacts and logs in: /artifacts/kv/splits/nodes=3/quiesce=true/run_1
(cluster.go:1933).Run: output in run_062148.310928853_n4_workload_run_kv: ./workload run kv --init --max-ops=1 --concurrency=192 --splits=300000 {pgurl:1-3} returned: context canceled
(monitor.go:127).Wait: monitor failure: monitor command failure: unexpected node event: 1: dead (exit status 137)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity added this to the 23.1 milestone Dec 14, 2022

blathers-crl bot added the T-kv KV Team label Dec 14, 2022

kvoli self-assigned this Dec 14, 2022

kvoli added a commit to kvoli/cockroach that referenced this issue Dec 14, 2022

kvserver: refactor replica load

b8813fb

wip resolves cockroachdb#87187 resolves cockroachdb#93579 Release note: None

pav-kv added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. A-kv-observability labels Dec 14, 2022

blathers-crl bot added the T-kv-observability label Dec 14, 2022

pav-kv removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Dec 14, 2022

kvoli mentioned this issue Dec 16, 2022

kvserver: refactor replica load #93823

Merged

craig bot closed this as completed in 6225ad6 Jan 4, 2023

exalate-issue-sync bot removed the T-kv KV Team label Jan 4, 2023

This was referenced Jan 4, 2023

roachtest: splits/largerange/size=32GiB,nodes=6 failed [OOM] #94371

Closed

roachtest: kv/splits/nodes=3/quiesce=true failed #94909

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

roachtest: kv/splits/nodes=3/quiesce=true failed #93579

roachtest: kv/splits/nodes=3/quiesce=true failed #93579

cockroach-teamcity commented Dec 14, 2022 •

edited by exalate-issue-sync bot

Loading

kvoli commented Dec 14, 2022 •

edited

Loading

pav-kv commented Dec 14, 2022

kvoli commented Dec 14, 2022

pav-kv commented Dec 14, 2022

cockroach-teamcity commented Dec 15, 2022

kvoli commented Dec 15, 2022

cockroach-teamcity commented Dec 18, 2022

kvoli commented Dec 19, 2022

cockroach-teamcity commented Dec 22, 2022

cockroach-teamcity commented Dec 23, 2022

cockroach-teamcity commented Dec 25, 2022

cockroach-teamcity commented Dec 26, 2022

cockroach-teamcity commented Dec 27, 2022

kvoli commented Dec 28, 2022 •

edited

Loading

cockroach-teamcity commented Dec 29, 2022

cockroach-teamcity commented Dec 30, 2022

cockroach-teamcity commented Jan 2, 2023

cockroach-teamcity commented Jan 3, 2023

cockroach-teamcity commented Jan 4, 2023

roachtest: kv/splits/nodes=3/quiesce=true failed #93579

roachtest: kv/splits/nodes=3/quiesce=true failed #93579

Comments

cockroach-teamcity commented Dec 14, 2022 • edited by exalate-issue-sync bot Loading

kvoli commented Dec 14, 2022 • edited Loading

pav-kv commented Dec 14, 2022

kvoli commented Dec 14, 2022

pav-kv commented Dec 14, 2022

cockroach-teamcity commented Dec 15, 2022

kvoli commented Dec 15, 2022

cockroach-teamcity commented Dec 18, 2022

kvoli commented Dec 19, 2022

cockroach-teamcity commented Dec 22, 2022

cockroach-teamcity commented Dec 23, 2022

cockroach-teamcity commented Dec 25, 2022

cockroach-teamcity commented Dec 26, 2022

cockroach-teamcity commented Dec 27, 2022

kvoli commented Dec 28, 2022 • edited Loading

cockroach-teamcity commented Dec 29, 2022

cockroach-teamcity commented Dec 30, 2022

cockroach-teamcity commented Jan 2, 2023

cockroach-teamcity commented Jan 3, 2023

cockroach-teamcity commented Jan 4, 2023

cockroach-teamcity commented Dec 14, 2022 •

edited by exalate-issue-sync bot

Loading

kvoli commented Dec 14, 2022 •

edited

Loading

kvoli commented Dec 28, 2022 •

edited

Loading