roachtest: hotspotsplits/nodes=4 failed

[cockroach-teamcity]

(roachtest).hotspotsplits/nodes=4 failed on master@c36f7cbfe08d48025540d0af62637a2fdd1f7d89:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200403-1848230/hotspotsplits/nodes=4/run_1
	hotspotsplits.go:83,hotspotsplits.go:104,test_runner.go:753: range size 1.5 GiB exceeded 1.5 GiB
		main.registerHotSpotSplits.func1.2
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/hotspotsplits.go:67
		github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		runtime.goexit
			/usr/local/go/src/runtime/asm_amd64.s:1357

More

Artifacts: /hotspotsplits/nodes=4

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

[nvanbenschoten]

@ajwerner this was run on a SHA that includes #46863, so it seems likely that it's related. Do you mind taking a look?

[cockroach-teamcity]

(roachtest).hotspotsplits/nodes=4 failed on master@82fec00c83d4bfe35b906264ccb568568cec15b7:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200404-1851668/hotspotsplits/nodes=4/run_1
	hotspotsplits.go:83,hotspotsplits.go:104,test_runner.go:753: range size 1.5 GiB exceeded 1.5 GiB
		main.registerHotSpotSplits.func1.2
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/hotspotsplits.go:67
		github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		runtime.goexit
			/usr/local/go/src/runtime/asm_amd64.s:1357

More

Artifacts: /hotspotsplits/nodes=4

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

[cockroach-teamcity]

(roachtest).hotspotsplits/nodes=4 failed on master@b1a0b989bbfef500075a485edc762fe42ca7b32a:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200405-1853455/hotspotsplits/nodes=4/run_1
	hotspotsplits.go:83,hotspotsplits.go:104,test_runner.go:753: range size 1.5 GiB exceeded 1.5 GiB
		main.registerHotSpotSplits.func1.2
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/hotspotsplits.go:67
		github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		runtime.goexit
			/usr/local/go/src/runtime/asm_amd64.s:1357

More

Artifacts: /hotspotsplits/nodes=4

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

[cockroach-teamcity]

(roachtest).hotspotsplits/nodes=4 failed on master@beac4a53e0e2e2236eb5957f67abc1bf476ad1b6:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200406-1854541/hotspotsplits/nodes=4/run_1
	hotspotsplits.go:83,hotspotsplits.go:104,test_runner.go:753: range size 1.5 GiB exceeded 1.5 GiB
		main.registerHotSpotSplits.func1.2
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/hotspotsplits.go:67
		github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		runtime.goexit
			/usr/local/go/src/runtime/asm_amd64.s:1357

More

Artifacts: /hotspotsplits/nodes=4

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

[cockroach-teamcity]

(roachtest).hotspotsplits/nodes=4 failed on master@2032dafccfa311c7538960e974953cb9dc1d4e50:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200407-1857109/hotspotsplits/nodes=4/run_1
	hotspotsplits.go:83,hotspotsplits.go:104,test_runner.go:753: range size 1.5 GiB exceeded 1.5 GiB
		main.registerHotSpotSplits.func1.2
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/hotspotsplits.go:67
		github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		runtime.goexit
			/usr/local/go/src/runtime/asm_amd64.s:1357

More

Artifacts: /hotspotsplits/nodes=4

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

[ajwerner]

This test is absolutely failing due to #46271. The test runs 512KiB inserts at high concurrency. The backpressure check currently does not take into account in-flight requests. The tolerance added in that PR was 2MiB. With those setting, 4 concurrent requests to a range will lead to the tolerance being surpassed. My proposal to fix the problem is twofold:

1. Reduce the size of the inserts in the workload (perhaps to 256KiB).
2. Increase the kv.range.backpressure_byte_tolerance to something like 32MiB. 2MiB made some sense when it was the primary mitigation for the problem, but it no longer is.

@nvanbenschoten how do you feel about that?

[nvanbenschoten]

Thanks for taking a look. That proposal sounds good.

[cockroach-teamcity]

(roachtest).hotspotsplits/nodes=4 failed on master@10b3eeebba4ee1957b6f0c97fd64d972775e57ef:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200408-1860251/hotspotsplits/nodes=4/run_1
	hotspotsplits.go:83,hotspotsplits.go:104,test_runner.go:753: range size 1.5 GiB exceeded 1.5 GiB
		main.registerHotSpotSplits.func1.2
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/hotspotsplits.go:67
		github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		runtime.goexit
			/usr/local/go/src/runtime/asm_amd64.s:1357

More

Artifacts: /hotspotsplits/nodes=4

See this test on roachdash
_{powered by pkg/cmd/internal/issues}