spanconfig: add mechanism to alter the buffer limit used by the KVSubscriber

[arulajmani]

Is your feature request related to a problem? Please describe.

The KVSubscriber uses a rangefeedcache.Watcher over system.span_configurations to construct and incrementally maintain in-memory span configuration state. We limit this watcher's buffer size statically, here:

cockroach/pkg/server/server.go

Lines 547 to 555 in 2867db0

	spanConfig.subscriber = spanconfigkvsubscriber.New(
	clock,
	rangeFeedFactory,
	keys.SpanConfigurationsTableID,
	1<<20, /* 1 MB */
	fallbackConf,
	spanConfigKnobs,
	)
	}

We coarsely estimate each row in system.span_configurations to be 5KB. Given this, limiting the buffer size to 1MB seems sane for incremental updates. However, this limit is also enforced when performing an initial scan of the table as well, which is no good. We'll have a separate issue to fix that, but independent of it, it still seems prudent to have a mechanism to be able to alter the 1MB limit for the KVSubscriber's buffer in the wild as an escape hatch.

Describe the solution you'd like

Let's introduce an env var that dictates the buffer size to use when initializing the KVSubscriber. We could introduce a cluster setting as well, though that might be a bit more involved.

cc @ajwerner @irfansharif

Jira issue: CRDB-13643

[irfansharif]

We use the underlying rangefeedcache library at various places now, not just the KVSubscriber. Do we want knobs for each usage? Or are the other initial table scans not as worrying?

Let's introduce an env var that dictates the buffer size to use when initializing the KVSubscriber. We could introduce a cluster setting as well, though that might be a bit more involved.

Given the KVSubscriber is a per-store thing, probably we want an the cluster setting. Changing out the env-var cluster wide would require a full cluster restart -- not the best during incidents that want such a thing.

[ajwerner]

Let's set out to make the buffer size dynamic in the rangefeedcache library.

[irfansharif]

I'm still planning to make this limit dynamic and backport, but with #77687 this issue is no longer a release blocker. Concretely: running into buffer limits during the initial scan is unrecoverable, which #78148 fixes. Running into buffer limits during the incremental phase is entirely recoverable with a retry, which this issue would help reduce the occurrence of in practice.

[github-actions]

We have marked this issue as stale because it has been inactive for
18 months. If this issue is still relevant, removing the stale label
or adding a comment will keep it active. Otherwise, we'll close it in
10 days to keep the issue queue tidy. Thank you for your contribution
to CockroachDB!