feat(job): add removeDependencyOnFail option

[roggervalf]

ref #800

[manast]

Looks good, some small comments.

[manast]

Maybe a comment here clarifying that these fields are the ones stored in Redis and thus with smaller keys for compactness.

[manast]

Maybe this method should be private since it has no use outside of this class.

[manast]

Works for now but I guess in the future we will have a map object for translating all the options from Redis representation to library representation.

[roggervalf]

yes, this is the beginning of it

[manast]

hmm, shouldn't "activeParent" variable name be called instead something like: "isParentWaiting" ?

[manast]

could this function be renamed to "moveParentToWaitifNeeded" ? seems like that is what it does, "update" it is a big vague and makes it more difficult to understand.

[roggervalf]

sounds good to me

[M3talen]

Dear @manast ,

I wanted to reach out and express my appreciation for the work you have done on the this project. I have been using the project for a few projects, and I think it is a fantastic resource.

I was wondering if you would be open to considering the addition of this feature that I believe would greatly benefit the project. As right now there is no other way to complete a flow without removing the child manually. But the bigger problem for me is that removing child's from the array doesn't update the flow job.

Thank you for your time, and I look forward to hearing from you.

Sincerely,
@M3talen

[roggervalf]

hi @M3talen,
could you please elaborate a little bit your use case, now I'm thinking that if a job is not required to be completed for an specific flow, this should mean that the parent-child relationship is not needed in that case. Also as you mentioned, when a child is deleted, the parent should be moved to wait if this is the last pending child for that job as in https://github.com/taskforcesh/bullmq/blob/master/tests/test_flow.ts#L2480-L2525.

[M3talen]

hi @roggervalf
It might be a bit difficult to explain but ill try to show it off

1. in EngineQueue i create a new task (daily-flow)
2. that task creates a new flow with a parent called 'flow-job' in a new queue (FlowQueue)
3. as children i put tasks that need to be done daily in other queues
   3.1) Flow starts executing first children.
   3.2) job in EngineQueue gets put to completed. (here start the problems - if i wait for this job and a child fails it will stop procesing the queue)
   Now i don't care about what tasks fail but when one fails i would like to add a stacktrace/log/update data on the 'flow-job' so i know what failed but i want the 'flow-job' to be executed when all children either fail or complete.

Currently when a task fails (a child) the 'flow-job' is not executed until i delete it
even if the failed task fails all attempts and gets moved to failed it doesn't execute the 'flow-job'

I hope this explains a bit of my problem.

[roggervalf]

hi @M3talen, in your case you can use this function https://api.docs.bullmq.io/classes/FlowProducer.html#addBulk, for example:

const trees = await flow.addBulk([
        {
          name: 'flow-job',
          queueName: 'Flow-Queue',
          data: {},
          children: [
            {
              name,
              data: { idx: 0, foo: 'bar' },
              queueName: 'Flow-Queue',
            },
          ],
        },
        {
          name: 'non-dependent-child',
          queueName: 'Flow-Queue',
          data: {},
          children: [
            {
              name,
              data: { idx: 1, foo: 'baz' },
              queueName: 'Flow-Queue',
            },
          ],
        },
      ]);

so when you have children where you don't care if they complete or fail, those could be created apart of the main flow, with addBulk you can create all those flows together

[roggervalf]

in favor of #1953