Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: cdc/schemareg failed #36409

Closed
cockroach-teamcity opened this issue Apr 2, 2019 · 2 comments
Closed

roachtest: cdc/schemareg failed #36409

cockroach-teamcity opened this issue Apr 2, 2019 · 2 comments
Assignees
@danhhz
Labels
C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot.
Milestone
19.1

Comments

@cockroach-teamcity
Copy link
Member

SHA: https://github.com/cockroachdb/cockroach/commits/2851c7d56ee4966109691b5c48c73ec8d4cc9847

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=cdc/schemareg PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1215354&tab=buildLog

The test failed on master:
	cdc.go:451,cdc.go:607,test.go:1226: expected
		{"updated":{"string":""},"after":{"foo":{"a":{"long":1}}}}
		{"updated":{"string":""},"after":{"foo":{"a":{"long":2},"b":{"string":"2"}}}}
		{"updated":{"string":""},"after":{"foo":{"a":{"long":3},"b":{"string":"3"},"c":{"long":3}}}}
		{"updated":{"string":""},"after":{"foo":{"a":{"long":1},"c":null}}}
		{"updated":{"string":""},"after":{"foo":{"a":{"long":2},"c":null}}}
		{"updated":{"string":""},"after":{"foo":{"a":{"long":3},"c":{"long":3}}}}
		{"updated":{"string":""},"after":{"foo":{"a":{"long":4},"c":{"long":4}}}}
		{"updated":{"string":""},"after":{"foo":{"a":{"long":1},"c":null}}}
		{"updated":{"string":""},"after":{"foo":{"a":{"long":2},"c":null}}}
		{"updated":{"string":""},"after":{"foo":{"a":{"long":3},"c":{"long":3}}}}
		{"updated":{"string":""},"after":{"foo":{"a":{"long":4},"c":{"long":4}}}}
		
		got
		{"updated":{"string":""},"after":{"foo":{"a":{"long":1}}}}
		{"updated":{"string":""},"after":{"foo":{"a":{"long":2},"b":{"string":"2"}}}}
		{"updated":{"string":""},"after":{"foo":{"a":{"long":3},"b":{"string":"3"},"c":{"long":3}}}}
		{"updated":{"string":""},"after":{"foo":{"a":{"long":1},"c":null}}}
		{"updated":{"string":""},"after":{"foo":{"a":{"long":2},"c":null}}}
		{"updated":{"string":""},"after":{"foo":{"a":{"long":3},"c":{"long":3}}}}
		{"updated":{"string":""},"after":{"foo":{"a":{"long":1},"c":null}}}
		{"updated":{"string":""},"after":{"foo":{"a":{"long":2},"c":null}}}
		{"updated":{"string":""},"after":{"foo":{"a":{"long":3},"c":{"long":3}}}}
		{"updated":{"string":""},"after":{"foo":{"a":{"long":4},"c":{"long":4}}}}
		
	cluster.go:1329,cdc.go:649,asm_amd64.s:523,panic.go:397,test.go:774,test.go:764,cdc.go:451,cdc.go:607,test.go:1226: test already failed
@cockroach-teamcity cockroach-teamcity added this to the 19.1 milestone Apr 2, 2019
@cockroach-teamcity cockroach-teamcity added C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. labels Apr 2, 2019
@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/877ebd1ece299b9ee621aa0d091657621593d844

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=cdc/schemareg PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1215372&tab=buildLog

The test failed on release-19.1:
	cdc.go:451,cdc.go:607,test.go:1226: expected
		{"updated":{"string":""},"after":{"foo":{"a":{"long":1}}}}
		{"updated":{"string":""},"after":{"foo":{"a":{"long":2},"b":{"string":"2"}}}}
		{"updated":{"string":""},"after":{"foo":{"a":{"long":3},"b":{"string":"3"},"c":{"long":3}}}}
		{"updated":{"string":""},"after":{"foo":{"a":{"long":1},"c":null}}}
		{"updated":{"string":""},"after":{"foo":{"a":{"long":2},"c":null}}}
		{"updated":{"string":""},"after":{"foo":{"a":{"long":3},"c":{"long":3}}}}
		{"updated":{"string":""},"after":{"foo":{"a":{"long":4},"c":{"long":4}}}}
		{"updated":{"string":""},"after":{"foo":{"a":{"long":1},"c":null}}}
		{"updated":{"string":""},"after":{"foo":{"a":{"long":2},"c":null}}}
		{"updated":{"string":""},"after":{"foo":{"a":{"long":3},"c":{"long":3}}}}
		{"updated":{"string":""},"after":{"foo":{"a":{"long":4},"c":{"long":4}}}}
		
		got
		{"updated":{"string":""},"after":{"foo":{"a":{"long":1}}}}
		{"updated":{"string":""},"after":{"foo":{"a":{"long":1},"c":null}}}
		{"updated":{"string":""},"after":{"foo":{"a":{"long":2},"b":{"string":"2"}}}}
		{"updated":{"string":""},"after":{"foo":{"a":{"long":2},"c":null}}}
		{"updated":{"string":""},"after":{"foo":{"a":{"long":3},"b":{"string":"3"},"c":{"long":3}}}}
		{"updated":{"string":""},"after":{"foo":{"a":{"long":3},"c":{"long":3}}}}
		{"updated":{"string":""},"after":{"foo":{"a":{"long":1},"c":null}}}
		{"updated":{"string":""},"after":{"foo":{"a":{"long":2},"c":null}}}
		{"updated":{"string":""},"after":{"foo":{"a":{"long":3},"c":{"long":3}}}}
		{"updated":{"string":""},"after":{"foo":{"a":{"long":4},"c":{"long":4}}}}
		
	cluster.go:1329,cdc.go:649,asm_amd64.s:523,panic.go:397,test.go:774,test.go:764,cdc.go:451,cdc.go:607,test.go:1226: test already failed

@danhhz
Copy link
Contributor

danhhz commented Apr 2, 2019

Not only are they in a different order, but they also have a different set of duplicates. I was hoping to get away with not doing the assertions based on sorting and uniquing (which we do in all the unit tests), but looks like we'll have to do it here at well.

danhhz added a commit to danhhz/cockroach that referenced this issue Apr 2, 2019
@danhhz
roachtest: handle duplicates in cdc/schemareg
48828d1 
There are various internal races and retries in changefeeds that can
produce duplicates. This test is really only to verify that the
confluent schema registry works end-to-end, so do the simplest thing and
sort + unique the output.

Closes cockroachdb#36409

Release note: None
@danhhz danhhz mentioned this issue Apr 2, 2019
Merged
craig bot pushed a commit that referenced this issue Apr 2, 2019
@lucy-zhang @danhhz
Merge #36403 #36436
eef79e8 
36403: storage: rate-limit AddSST requests r=lucy-zhang a=lucy-zhang

We've been seeing extremely high latency for foreground traffic during bulk
index backfills, because AddSST requests into non-empty ranges can be
expensive, and write requests that are queued behind an AddSST request for an
overlapping span can get stuck waiting for multiple seconds. This PR limits the
number of concurrent AddSST requests for a single store, determined by a new
cluster setting, `kv.bulk_io_write.concurrent_addsstable_requests`, to decrease
the impact of index backfills on foreground writes. (It also decreases the risk
of writing too many L0 files to RocksDB at once, which causes stalls.)

Fixes #36430

Release note (general change): Add a new cluster setting,
`kv.bulk_io_write.concurrent_addsstable_requests`, which limits the number of
SSTables that can be added concurrently during bulk operations.

36436: roachtest: handle duplicates in cdc/schemareg r=nvanbenschoten a=danhhz

There are various internal races and retries in changefeeds that can
produce duplicates. This test is really only to verify that the
confluent schema registry works end-to-end, so do the simplest thing and
sort + unique the output.

Closes #36409

Release note: None

Co-authored-by: Lucy Zhang <[email protected]>
Co-authored-by: Daniel Harrison <[email protected]>
@craig craig bot closed this as completed in #36436 Apr 2, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@danhhz danhhz

Labels
C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot.
Projects
None yet
Milestone
19.1
Development

No branches or pull requests
2 participants
@danhhz @cockroach-teamcity