Once-off scheduled Workflow execution

[byrneo]

Summary

There are times where you may wish to schedule just a single Workflow execution at a specific, future point in time. E.g. I may wish to execute Workflow-A on Wed 1 Mar 2023 14:05:30 GMT, Workflow-B on Fri 3 Mar 2023 20:10:05 GMT etc

Cron Workflows are a great fit for recurring Workflow executions e.g. once every hour, week etc etc but, by design, don't enable a single invocation for a specific future date/time.

What is being suggested here is a Workflow type that would mimic the feature of linux's at command where you can schedule a job for execution at a future point in time.

Use Cases

Execute Workflows, individually, on a scheduled basis with no recurrence.

---

Message from the maintainers:

Love this enhancement proposal? Give it a 👍. We prioritise the proposals with the most 👍.

[nakamorichi]

How about using some notification engine (e.g. Cronicle) for sending scheduled notifications to Argo Events and then trigger Workflows from there?

[nakamorichi]

Not sure how difficult it would be to implement such functionality into Workflow. Having scheduled execution for Workflow, would be nice to have feature, though.

[caelan-io]

Another option until this is a native feature in Argo WF: create a plugin step that uses a script or notification engine integration to schedule the workflow:
https://argoproj.github.io/argo-workflows/plugins/

a plugin just responds to RPC HTTP requests
https://argoproj.github.io/argo-workflows/executor_plugins/

[eduardodbr]

Hi!
I would also need to run workflows on a schedule so did a bit of research on how this could be implemented using cron workflows:

• Define the year on the cron expression: Currently argo workflows uses https://github.com/robfig/cron to schedule cron jobs but this lib does not support setting the year. There is a PR opened for it, but there is no response from the maintainer. The project seems stale for the last 2.5 years.
• Create a new config field to set the max number of runs for a CronWorkflow: the controller would stop scheduling the cronjob once it reached the configured max runs. This seems fairly simple to implement.

Other than this I can only see a new controller being created.

Let me know your thoughts on this. I am available to contribute to this.

[tico24]

This is a very commonly-requested feature in Slack so I'd love to see it. My vote is for your second approach, setting a maxRuns value on a cronWorkflow.

[caelan-io]

I also have seen this requested commonly. Scheduling a one-off workflow is a common feature of other workflow orchestrators, like Airflow, so I would propose we include this in a future v3.5 release.

In addition to the OP's feature request on Slack, here's another recent Slack request for this feature:

Is it possible to schedule a workflow for a later date and have it run only once?

@terrytangyuan - what do you think? would this approach work for solving this feature gap? If so, @eduardodbr mentioned he could contribute it:

Create a new config field to set the max number of runs for a CronWorkflow: the controller would stop scheduling the cronjob once it reached the configured max runs. This seems fairly simple to implement.

[eduardodbr]

It would also fix #7201

[Joibel]

The idea of a workflow that has maxRuns != infinity or maxRuns != 1 is a bit odd for the way I think of workflows. Here I am assuming this will be implemented using the status section of a CronWorkflow recording number of attempted runs as I can't think of an alternative. If I'm wrong then please correct me.

CronWorkflows are thrown into your cluster to do repeating work. What's the correct behaviour if maxRuns = 5 and after 3 runs the workflow gets altered.

The fact that a replace of a CronWorkflow will cause us to restart from zero up to maxRuns is particularly unhappy as I don't really want to be backing up my CronWorkflow statuses to guarantee I never exceed maxRuns.

Instead I'd advocate a design for a once only scheduled future run of a workflow.

The design would be akin to an existing Workflow, but with a date/time when you'd like it to happen. deault: It would also have an acceptable time window, say 5 minutes, in which it is acceptable to start the Workflow after this time just in case there is a reason why it wasn't possible to run it dead on the desired time.

This prevents the problem of not really knowing what was intended by a CronWorkflow that's been sat inside my cluster for 3 years with a maxRuns of 5 which all expired 2 years ago. Can I delete it, should I? Should I restore it to my replacement cluster?

I prefer the original approach of just a one off scheduled execution. If you inject that after that date, it doesn't run. This does need the year adding to cron though to make sense.

[caelan-io]

This does need the year adding to cron though to make sense.

This seems to be the key design assumption and unfortunate situation of the unmaintained library dependency @eduardodbr highlighted: https://github.com/robfig/cron - It doesn't seem the library's maintainer would merge this change. Perhaps others have ideas about how to account for this blocker.

In the meantime, it seems to me like maxRuns + good docs to inform users what happens with the CronWorkflows when the maxRuns aren't ever exceeded would be a reasonable path forward from a community contribution & maintenance standpoint.

[terrytangyuan]

Would this be achievable via Calendar event source with Argo Events? https://argoproj.github.io/argo-events/eventsources/setup/calendar/

[tico24]

Calendar events also uses cron, so no.

Users also won't want to install a whole other product just for this one feature.

[terrytangyuan]

One wild idea: in the CronWorkflow spec, define a last step to suspend the CronWorkflow itself. There should be other similar ways to achieve this.

[tico24]

Then you have a hanging workflow and new ones arriving on the cron schedule.

Edit: ah. Suspend the cron, not the workflow. Still hacky and won't work with argocd unless you start playing with ignoreDifferences. Surely simply adding the much-requested feature is easiest here?

[terrytangyuan]

Then deleting itself as the last step instead of suspending?

[tico24]

This feels horrifically hacky. And goes against gitops principals.

[terrytangyuan]

More generalized idea #7291. Also related #7201 #5659

[eduardodbr]

@terrytangyuan those are related but not really the same. However, I believe both could add value. btw, is there any reason #5662 was never merged?

[terrytangyuan]

Yes I agree they are both valuable so I'll be on the look for proposals that are generalized and extensible instead of one-off solution like maxRuns.

I was working on other high priority tasks so I stopped working on #5662 but feel free to take it over if anyone has time.

[wesleyscholl]

A scheduled cron to run once would be very useful. Especially when scheduling tasks after business hours.

Is it possible to use history limits as a workaround?

successfulJobsHistoryLimit: 1
failedJobsHistoryLimit: 1 

[nakamorichi]

Instead of specifying a single scheduled date, I would prefer to be able to pass a list of dates in case I want to execute a workflow on multiple irregular dates. Problem with cron is that it expects regular occurrence, so basically we would need a mechanism to support irregular occurrences.

e.g. add field irregularSchedule and make it possible to set irregular schedule dates in addition to cron pattern.

Considering all the irregular patterns, implementing feature that would satisfy everyone, is challenging. As having worked with custom calendar functionality in the past, event recurrence and exceptions are huge pain to implement.

In order to avoid having to modify cron pattern, would need to have also a way to add exceptions, e.g. "don't execute between datetimes X and Y". And during that period could utilize irregularSchedule for setting different execution times temporarily, in case there's e.g. some major company event occurring, and after that want to return to the regular schedule.

In Google Calendar, we could remove a single instance from recurring event and create one-off event to achieve such, but not sure how similar could be implemented in Argo WF.

Maybe using separate calendar system is the best after all in case need to handle exceptions to regular cron schedule and one-off workflows?

[sarabala1979]

It is a good feature to add to the cron workflow. It may have a flag to say count , or recurring.

[terrytangyuan]

More generalized issue #7291

maxRuns can be depending on the status of workflows:
e.g. maxSuccessRuns - depending on status

[caelan-io]

Hi @eduardodbr - We discussed today in the Contributor Meeting. Would you be available to create a short enhancement proposal PR that links this issue? We'll then do a final round of feedback on the solution design. (example proposals: 1, 2)

More color on @terrytangyuan 's comment above:

• If possible to created a conditional variable for maxRuns (a la maxSuccessRuns) based on the status of the workflow, that would address use cases we have seen before via Circuit Breakers for CronWorkflows #7291.

[sarabala1979]

I like @terrytangyuan idea

RunStrategy:
  maxRun:
  maxSuccess:
  maxFailures:
  expression:

[nakamorichi]

Took a look at glance, but would it be possible to support both regular and irregular execution without having to duplicate execution logic (i.e. Workflow / WorkflowTemplate / CronWorkflow / etc.)?

In any case, there should be a simple mechanism for creating scheduled irregular workflow executions, and ideally it should be easy to figure out both regular and irregular execution schedule. Like in calendar APIs, it should be possible to query scheduled workflow executions via single API.