Batch job canceling ?

[Pascal-Delange]

Hi,

I have a usecase where I run a (possibly) large batch job, composed of many unrelated work units to run. "Many" could be, say, 100ks or millions. Those jobs, however, would run infrequently.

I like to use a task queue for the individual tasks, for retries and scheduling and so on, but I also need a separate routine (which I implemented as a separate job kind) that performs the status management of the overall task: updating progress, timeout, etc.
Now, I want to make it so that, if the "outer" ("big") job is marked as canceled, I can pre-emptively cancel all the "inner" (small) jobs - there may be quite a lot of those inner jobs, say, also of the order of magnitude of 100ks or more.

The goal would be to avoid having to wait possibly hours until all the remaining tasks get executed just to notice that they have nothing left to do and return, possibly delaying other tasks (that are now more relevant) for the same customer.

I've been trying to implement this with River, because I don't want a separate Redis piece of infra (for now), and I'm switching from a more "naive" implementation where individual tasks are attempted sequentially and the big job stops at the first error encountered.

I was wondering what the best way to do this would be. The best options I see are:

• using the river API to list jobs matching some metadata, and use JobCancel[Tx] to cancel all of them individually
• considering the river tables are in my db, just run a raw sql "delete" (or "update"?) statement on all the remaining rows

NB: this could totally happen asynchronously in a best effort manner.

Does the idea seem weird, and what do you think would be the best option ?
Thanks !