Bulk Workflow Manager not functional

[peter-g-ensemble]

Required Information

• AEM Version: 6.5
• ACS AEM Commons Version: 4.1.0
• Reproducible on Latest? yes

I followed the tutorial for the Bulk Workflow Manager v2 and was never able to actually get a functioning execution. It hangs at "Infinity years" remaining and never completes a single asset. I've linked a screenshot below. Resetting AEM multiple times had no effect. Workflows run fine when I do the limit of 15 from inside the touch UI. This issue happens on the bulk manager when the payload is 1 image or my entire dam folder. Hardware is a top of the line 2018 macbook pro.

https://imgur.com/a/t7BYYOP

Let me know if you have any ideas as to why this is happening.

Thanks,

Peter

[davidjgonzalez]

Hi Peter! Which WF runners exhibit this behavior?

[peter-g-ensemble]

@davidjgonzalez Hi David, its DAM Update Asset. I'm doing some tests for a future migration which will be at a much larger scale.

[davidjgonzalez]

Good to know - do you know what runner? Normal aem wf? Synthetic? Synthetic multi threaded?

[peter-g-ensemble]

@davidjgonzalez sorry my bad, its the normal AEM workflow.

[davidjgonzalez]

shoot! That’s the hardest one to debug :)

That said - if you can run it as synthetic that’s highly recommend from a performance POV. Iirc you can run as synthetic unless you’re running dynamic media (which invoked an external wf process step).

[peter-g-ensemble]

@davidjgonzalez Synthetic w/ FAM worked great. Will just avoid using normal AEM workflow I guess!

[davidjgonzalez]

I’ll still check it out ... btw if it says “synthetic w FAM” you’re running a pretty old version, no? Pretty sure it’s changed to “synthetic workflow (multi threaded)” since “FAM” is a ~meaningless term

[peter-g-ensemble]

@davidjgonzalez I think it just needs to be relabeled in the execution summary page. I'm definitely on 6.5 and 4.1.0. t's labeled as "Synthetic workflow (multi threaded)" in the dropdown list when creating the bulk workflow, but it shows "Synthetic w/ FAM" on the execution summary page.

[davidjgonzalez]

Gotcha! I’ll update that - thanks!

[davidjgonzalez]

@peter-g-ensemble I was able to reproduce this in AEM 6.5 easily. the AEM WF when executed (even on 1 payload!!!) throws all sorts of Oak conflicts and fails.. I'm not sure why this is happening and TBH won't have time to look into it and fix in the foreseeable future.

I quickly ported the Synthetic WF processing over to #1954 and hopefully, we can deprecate/remove the BWM.

[vesankar]

Hi Team,
I am facing the same issue in DAM UPDATE ASSET and my customer is using S7.
Can you use Synthetic multi threaded option? Please suggest.

Regards,
Venkatesh S

[davidjgonzalez]

@vesankar The DM steps do not work in synthetic wf as they are external WF process steps. If you are < 6.5 you could try using bulk workflow manager with the “aem workflow runner” however something has changed in how they work that Bulk workflow manager broke using them, so YMMV.

I have no time to look Into the aem workflow runner support issue further (did once but the reason for failure is not obvious) - if you want to poke around and PR a fix that would be super though!

[vesankar]

Thanks David.
We are using the OOTB DAM UPDATE ASSET workflow and using S7.
hope using synthetic would impact S7 integration, only if you configure to publish immediately.

if you configure to publish to S7 on activation, then this would not have any impact, right?
Plz confirm.

[davidjgonzalez]

@vesankar AFAIK the S7 Workflow step is a WorkflowExternalProcess step, which Synthetic WF cannot handle (since WorkflowExternalProcess steps sit there and "wait" for a response from the external service). Im not sure how publish immediately might differ, so I can't confirm.

From 6.5.0:

I would review all the Steps in Dam Update Asset WF you need to have run and make sure they're all WorkflowProcess types. If you can process the non-WorkflowProcess steps in some other way, and then before/after use Bulk synthetic to process the compatible steps, that might work.

[vesankar]

Thnx David.
We tried running the Bulk workflow using Synthetic multi-thread for DAM UPDATE ASSET.
On click of 'Start Bulk Workflow' button, we see the screen showing the status as completed,
But, actually it fails to process the asset.

It throws error like
javax.imageio.IIOException: No decoder available to load the image
org.apache.commons.imaging.ImageReadException: Can't parse this format.

Do you have any idea on this issue?
BTW, before running the bulk workflow, we have disabled the DAM UPDATE ASSET on create event, uploaded the assets.

[vesankar]

Hi David,

Please ignore my previous post.
I believe, it is the issue with specific input files.

In general, it works fine.

[vesankar]

Hi David,

Quick question.
I see that some of the assets show the asset state as PROCESSING, on running Bulk workflow manager on some assets.
But, this issue happens only when we run the workflow via this tool.
If i enable the DAM UPDATE ASSET workflow and upload the asset manually, asset gets processed completely and show as PROCESSED.

Do you have any thoughts on this?

[ravinderkp]

@davidjgonzalez I am also facing the same issue in AEM 6.4.2 and AEM 6.5 using the Runner as AEM Workflow for update_asset WF. But it is working fine with Runner as Synthetic Workflow w/ FAM and generating the renditions. Is there any ETA for the AEM Workflow issue?
Thanks
Ravinder

[davidjgonzalez]

@ravinderkp No ETA - I took a look at the cause wasn't obvious; For whatever reason, the AEM WF API's cant find the started WF by ID -- which makes me think perhaps something changed internally in AEM wrt to WF ID assignment for transient wf or something.

Long story short, I'd look at other approaches for apply non-synthetic WF to assets, as I'm not sure how to fix this and don't have time to look into it. If you can figure out the cause, i might be able to help provide guidance on how to fix it (or if the fix isnt worth implementing due to the required changes)

[badvision]

Check release 4.3.2 for the fix, released this week. Thanks for your patience in helping us understand this issue @ravinderkp and @peter-g-ensemble ... Much thanks to @davidjgonzalez for taking this bug by the horns. 😄

[davidjgonzalez]

It would be great if someone could smoke test the DMS7 workflow step w this - I think? The product started doing batch queuing or something under the covers. It should work since BWM just runs through transient and doesn’t really know what their state is. So I don’t see why it wouldn’t work, but didn’t have time to test against the latest dms7 func either

[ravinderkp]

@davidjgonzalez I did the testing today. with 4.3.2 on AEM 6.5. This is still showing Projected Time Remaining | Infinity years. But the improvement is it has generated the renditions and not throwing errors in the logs. After 11 minutes it was completed for 2 images .
Test Data
type=dam:Asset
path=/content/dam/we-retail/en/activities/hiking

Workflow Model | /var/workflow/models/dam/update_asset

[davidjgonzalez]

@ravinderkp I believe it should've placed all the assets under /content/dam/we-retail/en/activities/hiking in transient WF, which really just means Sling Jobs were created for them in batches per interval.

The time to complete is sort of meaningless since BWM cant tell when a Transient WF is complete.

[ravinderkp]

@davidjgonzalez Thanks for the response and efforts to resolve this issue :).

[davidjgonzalez]

@ravinderkp how are you measuring "2 images are completed"? and what parameters did you use to start it up?

You could crank up DEBUG log on com.adobe.acs.commons.workflow.bulk.execution as well to see if it gives more insights.