Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sql: maintain a singleton zcfg reconciliation task for sql tenants using the jobs framework #66680

Closed
arulajmani opened this issue Jun 21, 2021 · 1 comment · Fixed by #66806
Closed

sql: maintain a singleton zcfg reconciliation task for sql tenants using the jobs framework #66680

arulajmani opened this issue Jun 21, 2021 · 1 comment · Fixed by #66806
Assignees
@arulajmani
Labels
A-multitenancy Related to multi-tenancy A-zone-configs C-investigation Further steps needed to qualify. C-label will change.

Comments

@arulajmani
Copy link
Collaborator

arulajmani commented Jun 21, 2021 •
edited by irfansharif
Loading

Part of #66348. We want to ensure only a single reconciliation loop (promoting zcfgs to scfgs) is running for a given tenant across all its pods. We could use the jobs infrastructure for this; it would also provide a convenient way to checkpoint the reconciliation progress. This would be similar to how automatic stats collection works today.

Sketch

On startup, every sql pod will check to see if a reconciliation job exists (is running, pending, or paused), and if it doesn't, it will queue one up. The reconciliation process differs from the usual jobs mold in a few small ways. For one, it's a system initiated process, and as such should be non-cancellable. Additionally, it should never never enter the "terminal states" (succeeded/failed) that jobs do. It should perpetually stay in the "running" state, reacting to rangefeed events and reconciling as required.
@arulajmani arulajmani added C-investigation Further steps needed to qualify. C-label will change. A-zone-configs A-multitenancy Related to multi-tenancy labels Jun 21, 2021
@arulajmani arulajmani self-assigned this Jun 21, 2021
arulajmani added a commit to arulajmani/cockroach that referenced this issue Jun 24, 2021
@arulajmani
zcfgreconciler: setup non-cancellable, singleton reconciliation job
d1c2740 
This patch introduces the `ZcfgReconciliationManager` which is
responsible for ensuring one and only one zone config reconciliation
job exists. Every SQL pod creates one of these on startup. On startup,
the manager checks to see if a `ZcfgReconciliationJob` exists.
If it doesn't, it starts one up and marks it non-cancellable.

This patch creates and assigns a synthetic jobID to the
ZcfgReconciliationJob to ensure singleton semantics. The job resumer,
for now, is a no-op.

References cockroachdb#66680

Release note: None
@irfansharif irfansharif linked a pull request Jun 25, 2021 that will close this issue
[prototype] zcfgreconciler: setup non-cancellable, singleton reconciliation job #66806
Merged
@irfansharif
Copy link
Contributor

Weird, was supposed to have auto-closed with #66806.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@arulajmani arulajmani

Labels
A-multitenancy Related to multi-tenancy A-zone-configs C-investigation Further steps needed to qualify. C-label will change.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[prototype] zcfgreconciler: setup non-cancellable, singleton reconciliation job arulajmani/cockroach
2 participants
@arulajmani @irfansharif