Access kolibri's job storage database to reconcile tasks

[bjester]

Summary

• Implements tracking of the Work Manager process, thread, and request IDs in the database: job_storage.jobs.worker_*
• Small simplifications performed on existing code
• Refactors task request and query building out of Task.java into a shared utility
• Adds SQLite interface for reading and updating a database
• Adds SQLite implementation for interacting with select few columns on job_storage.jobs table
• Adds Sentinel class which can cross reference jobs in Kolibri's job_storage database with Work Manager's tasks
• Adds Reconciler class which can process results from Sentinel and re-queue tasks for any issues observed
• Adds Task.reconcile static method that uses Sentinel and Reconciler to reconcile tasks
• Adds a ReconcileWorker which processes task reconciliation

WIP / TBD

• The Reconciler implements exclusive locking, as I intended for reconciliation to be able to run both in a task and at startup of an app process, but currently it's only running in the task, pending results of QA
• The Reconciler implements a database transaction, but that is currently commented out, as it encountered a 'database is locked' error too many times

[rtibbles]

This is all making sense to me so far, even if the precise details escape or confuse me on occasion!

[rtibbles]

I definitely find reading overloaded implementation definitions a bit confusing - but also get why it's useful, I am just unused to it.

[bjester]

I definitely went a little overboard with the method overloading, but it is one of my favorite features. I like it because you can break down larger functions into smaller bits and tie them all together by overloading the name. It's also helpful for making optional parameters of course

[rtibbles]

I think I understand what is happening, and it makes sense to me. The sequential processing does seem like a sure fire way of stopping weird concurrency issues.

[rtibbles]

So, am I reading it correctly that now each job is being processed sequentially here, whereas previously there could be parallel processing of individual jobs?

[bjester]

Yep! The thenComposeAsync accepts a callback that itself returns another future, sort of like promise chaining, making everything process sequentially. This particular change seemed to provide the most improvement, and it makes the logs easier to read too.

My only worry was how we managed the DB connection without a try-with resources statement, but rather rely on the future completing in order to close the connection. CoPilot indicated there could be issues mixing futures and the try-with. I was also happy that CoPilot had nothing to say other than 'no issues could be detected and the code is well structured [....]'. With the addition of some timeouts, it should also prevent any stalling should it re-occur.

[bjester]

Just some minor simplification here as I learned more about how these futures and runnables work.

[rtibbles]

Squinting, I think I get the high level of the diff here. I am going to check it out locally, to read through the code to make sure I have a good picture in my head.

My hope is that some of this code can get trimmed somewhat once we move to chaquopy, so this diff is a bit of a bump that won't be completely permanent.

[rtibbles]

There are a couple of issues from manual QA to follow up on here - but it's not clear where the issues lie just yet, and this PR is big enough and robust enough for us to merge this and iterate further, here, and in Kolibri.