changefeedccl: flakey TestChangefeedRestartDuringBackfill

[irfansharif]

Describe the problem

I've seen TestChangefeedRestartDuringBackfill flake recently (CI failure here). It's possibly fallout from #73876. This issue tracks the investigation.

To Reproduce

dev test pkg/ccl/changefeedccl -f=TestChangefeedRestartDuringBackfill -v --timeout 10s --ignore-cache --show-logs

Jira issue: CRDB-12455

[andreimatei]

This failed for me twice. Last time here: https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_UnitTests/4147365?showRootCauses=true&expandBuildProblemsSection=true&expandBuildTestsSection=true

[irfansharif]

#75146 is a temporary bandaid for CI. Going through the bors queue over the last day looks like this test has been a frequent cause for failed builds. This is pretty easy to repro, especially with the following:

@@ -481,10 +485,10 @@ func withKnobsFn(fn updateKnobsFn) feedTestOption {
 var _ = withKnobsFn(nil /* fn */)

 func newTestOptions() feedTestOptions {
        // percentTenant is the percentange of tests that will be run against
        // a SQL-node in a multi-tenant server. 1 for all tests to be run on a
        // tenant.
-       const percentTenant = 0.25
+       const percentTenant = 1
        return feedTestOptions{
                useTenant: rand.Float32() < percentTenant,
        }

What I'm seeing happen is this test getting wedged here, reading from the changefeed:

cockroach/pkg/ccl/changefeedccl/changefeed_test.go

Line 4063 in bed5b79

assertPayloads(t, foo, []string{

The wedging only happens when there's a lot of concurrent split activity within the tenant's keyspace. The changefeed declared earlier in the test is declared over a specific table that forms a small portion of a larger single tenant range. With span configs, there's concurrent split queue activity splitting that range along the tenant's own table boundaries. There's something racey with the changefeed's internals and the range split activity that then causes the changefeed to get wedged. I'm not familiar with the internals here but going to start digging. @miretskiy, at first glance, does any of this ring a bell? Could you point me to the code that's responsible for the underlying rangefeed being robust to range splits? Has anything changed here recently? Is there anything special about declaring changefeeds over secondary tenants? Do we have tests that stress changefeed + concurrent split behavior?

To repro:

# Apply the patch above. With #75146 this will take 20s, each run taking <2s. 
# But by splitting in the tenant range, one run or another is bound to get wedged.
dev test pkg/ccl/changefeedccl -f=TestChangefeedRestartDuringBackfill -v --count 10 --timeout 30s --ignore-cache --show-logs

[blathers-crl]

cc @cockroachdb/cdc

[irfansharif]

PS: I'm not actively investigating this. Letting it sit on the CDC board to remind ourselves to address it during stability at the latest.