[CON-151] Refactor reconciliation queue

[theoilie]

Description

• Add state reconciliation queue skeleton
• Make state monitoring jobs return data about users instead of syncs and update-replica-set-operations
• Rename job processors to end with .jobProcessor.js
• Make minor miscellaneous test changes from previous feedback

The big change with this skeleton is that the monitoring queue outputs the building blocks instead of the actual potential sync ops and update replica set ops. Two other monitoring queue jobs will use those building blocks to separately determine syncs and reconfigs, which will be passed individually to the reconciliation queue. This decoupling also helps make _aggregateOps() easier to read (Linear card for that) and makes sense because determining syncs is much simpler and doesn't need all the complicated logic that determining reconfigs uses.

After this refactor there will be 4 reconciliation jobs (all part of the same queue but different job types, processor functions, and concurrencies):

1. Issue Sync Request Job: determines if a single user should be synced (based on clock value) and issues a request to the user's nodes to sync
2. Update Replica Set Job: performs a reconfiguration of a single user's replica set
3. Recurring Sync Job: triggered by Sync Request Job from another node. Performs the actual sync
4. Manual Sync Job: same as snapback's manual sync queue but as a job type within the larger Reconciliation Queue

The monitoring queue will wait for State Monitoring Jobs to complete and then use the output of those jobs to enqueue Sync Request Jobs and Update Replica Set Jobs for individual users.

Tests

I ran A up locally to make sure there were no new errors in the logs and verified that bull dashboard shows the rate limited monitoring jobs still running successfully with the updated output. New tests will come after functionality is reshuffled. Existing tests are updated to pass:

• StateMonitoringQueue.test.js
• monitorState.jobProcessor.test.js (renamed from processStateMonitoringJob.test.js)

How will this change be monitored? Are there sufficient logs?

There shouldn't be any new errors containing the following:

• processStateMonitoringJob ERROR
• StateMonitoringQueue ERROR
• StateReconciliationQueue ERROR

[SidSethi]

generally skeleton makes sense

we should maybe clarify the naming a bit to distinguish btwn outgoing sync request and the processing of a sync request. not sure of best approach
telling a node to sync against yourself vs another node teling you to sync against it vs a user telling you to sync from another node

not sure of best solution, but the naming currently doesn’t seem to do a great job of conveying the intent - wdyt

i really like the names you used for the queue processor types, very clear

[theoilie]

thanks for the review @SidSethi. I was thinking about the sync naming as well and didn't come to a great name. maybe just adding "outgoing" and "local" would help, although it would make the processor name long -- something like:

• processIssueOutgoingSyncRequestsJob.js to send sync requests to this node or other nodes
• processExecuteLocalSyncJob.js to execute (on this node) manual and recurring requests that come in from this node or other nodes

[SidSethi]

my impression was that each stateReconciliation job would handle a single user / operation? eg the final step of each stateMonitoringQueue job would be to enqueue jobs for every single operation by job type into stateREconciliation
it seems like currently each stateREcon job handles multiple ops?

otherwise looks fine but since this is mostly boilerplate/skeleton code, not a whole lot to review - generally all good

maybe easier to sync via slack quickly

[theoilie]

@SidSethi I thought about it a little more after we synced offline, and breaking apart finding syncs vs finding reconfigs into separate monitoring jobs makes sense to me. I'm holding off on putting that into this PR so I can flesh it out locally and make sure it looks reasonable (and just combine into 1 job if not). for now I just updated the recon jobs to only handle one user at a time

[SidSethi]

@SidSethi I thought about it a little more after we synced offline, and breaking apart finding syncs vs finding reconfigs into separate monitoring jobs makes sense to me. I'm holding off on putting that into this PR so I can flesh it out locally and make sure it looks reasonable (and just combine into 1 job if not). for now I just updated the recon jobs to only handle one user at a time

sounds good!

[SidSethi]

looks good! i don't see this as a particularly valuable change to merge in current state, vs waiting to fill in the skeleton more (even if not wired up) and merge together. your call tho
test coverage is nice

also pls update pr description with latest and greatest :)

[theoilie]

I'd rather get it merged so the next PR won't have as much bloat -- it's definitely going to be a big one. updated the PR description and merging once tests pass