JOIN task taking too long to execute

[shekarpcs]

We have join task in one of the big workflow which is taking longer to finish even the depedent tasks are completed.

please help what is causing conductor to execute join task so slow

[bjpirt]

We're also seeing this - we've implemented a basic test with a dynamic fork and after all of the sub tasks have completed, the Join always seems to take 31 seconds to complete, despite it having no real work to do.

[shekarpcs]

This is caused big issue in production , we had to roll back conductor version upgrade in high tensed situation.

could some one help on this. We are stuck in the middle.

[bjpirt]

Modifying the property conductor.app.systemTaskWorkerCallbackDuration enabled us to reduce this. We've set it to 1s and the JOIN completes almost immediately

[shekarpcs]

After setting conductor.app.systemTaskWorkerCallbackDuration=1s some of the workflows in heavy load staying in running state even after completing all task. surprisingly it is happening with same worklfows with join task

[shekarpcs]

Let me try this. Thanks a lot, it really helps.

…

On Thu, Jan 12, 2023, 9:41 AM Ben Pirt ***@***.***> wrote: Modifying the property conductor.app.systemTaskWorkerCallbackDuration enabled us to reduce this. We've set it to 1s and the JOIN completes almost immediately — Reply to this email directly, view it on GitHub <#3436 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AXQ2C7W547TNHMATK2AUFWTWSAJZ3ANCNFSM6AAAAAATXHHH2E> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[john-larson]

We are experiencing the same problem. And I have opened a new issue under the bugs section: #3472

[john-larson]

@shekarpcs @bjpirt Could you please share the latest version you are using that is free of this bug?

[bjpirt]

We've only just started using Conductor so haven't used a version without it - v3.13.2 exhibited the behaviour for us

[shekarpcs]

We used 3.4.1 before and it was working fine. Workflows are not set status to complete in high load also. We are struggling so much with this issue. We have no help from community.

…

On Tue, Jan 31, 2023, 4:48 AM john-larson ***@***.***> wrote: @shekarpcs <https://github.com/shekarpcs> @bjpirt <https://github.com/bjpirt> Could you please share the latest version you are using that is free of this bug? — Reply to this email directly, view it on GitHub <#3436 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AXQ2C7V6U46KHMDA3RHJ73LWVDNVDANCNFSM6AAAAAATXHHH2E> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[james-deee]

Yeah we're seeing the same thing. It sounds like it has to do with this change #3284

It's unforunate. We actually dont see this behavor in 3.11.3 either. But, it looks like the performance of this change is an unintended side effect? @apanicker-nflx do you have some insight here?

This is currently blocking us from upgrading Conductor because of the pressure it puts on the underlying DB, and because of how slow these JOINs are finishing.

[Jozefiel]

We had the same issue, but we limit the CPU in our compose file to only 2.
After changing to 4 CPUs, JOIN was much faster.
But previously was fast with 2 CPUs :(

[james-deee]

@apanicker-nflx just wanted to see if you have some insight here on this issue. It's blocking us from upgrading Conductor at the moment due to the performance regression of this change.

[shekarpcs]

I do not want you go through the same pain what we went through. .Replying quickly .

We had found following parameter which make conductor to execute tasks based on setting. It is not completing after 3seconds.

conductor.app.systemTaskWorkerCallbackDuration=3s

the default value is 30 seconds. Based on this it was taking 30 seconds to complete join task.

[james-deee]

@shekarpcs yeah we already changed it to that. But it A) puts an extreme load on the backing datastore and B) still takes 10s to process in our environment.

[shekarpcs]

We added horizontal pods for conductor to process normally. You can do that to increase performance

…

On Tue, Apr 18, 2023, 1:48 PM Jamie DeMichele ***@***.***> wrote: @shekarpcs <https://github.com/shekarpcs> yeah we already changed it to that. But it A) puts an extreme load on the backing datastore and B) still takes 10s to process in our environment. — Reply to this email directly, view it on GitHub <#3436 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AXQ2C7XCITYX47KPE2ITTKTXB3HYPANCNFSM6AAAAAATXHHH2E> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[james-deee]

Thanks @apanicker-nflx. I will attempt this in our version, and see if we get acceptable performance in our environement with this change.

[james-deee]

@apanicker-nflx just getting to this, but it looks like your instructions are missing other things. For example, it looks like JoinTaskMetadata isnt a class in this project. Also looks like WorkflowSystemTask is now a parameterized class in your example?

Any more guidance would be appreciated.

[manan164]

Hi @james-deee , Here is the sample changes. This is for reference purposes. I have tested this change and it works but I am yet to write the test for that.

[apanicker-nflx]

Thanks @manan164
Apologies @james-deee , the branch I pulled the changes from has other unrelated changes that crept through here. The PR above maps the changes correctly though.

[shekarpcs]

@apanicker-nflx @james-deee , are these changes in new version conductor?

We would like to upgrade to new version as soon as fix is available.

[VerstraeteBert]

We're still observing up to 30s delays after upgrading to 3.16.7, even after setting CallbackAfterSeconds to 1s for system tasks. Any idea what could still be going wrong?