Enabling variables to control job batch limits

[mramireztgtg]

We've been encountering recurring issues with some AWS Batch queues getting stuck in a Runnable state. This seems to be caused by resource availability constraints or potential misconfigurations in our workflows. To address this, I'm implementing a Job State Limit for these jobs in the metaflow-computation module. The AWS documentation allows us to extend the capabilities of this resource
Here’s the approach:

• Introduce an optional variable that allows us to configure the timeout and define the action to take when a job exceeds the allowed state duration.
  The benefits of this change are clear:
• It would prevent jobs from running excessively long (In our case we had a run 40+ hours over weekends), which has been blocking new executions and impacting production workloads.
• It addresses an issue several teams have mentioned in Slack threads, making it a valuable improvement for our broader community.

[delagoya]

This parameter is specific to Blocked Job Queue events

I would update the variables to follow the Batch API.

E.g. instead of job_state_limit_action_timeout it should be job_state_limit_action_max_time_seconds. The full set of variables should be:

• job_state_limit_action - type should be either string CANCEL or null
• job_state_limit_action_max_time_seconds - type number greater than 600
• job_state_limit_action_reason - type enum string, one of the valid reasons in the Batch documentation
• job_state_limit_action_state - type string, either RUNNABLE or null

Implementation note:

Since action and state each only allow a single value, it would be possible to omit them for now and just have max_time_seconds and reason but the API may change at a later date to add more possible values to these API parameters.

[delagoya]

Besides adding the variables to main.tf you will need to update the /modules/computation/batch.tf aws_batch_job_queue resource.

[mramireztgtg]

@delagoya agree
I've pushed the suggested changes

[delagoya]

Point to Batch docs for valid reasons

[delagoya]

This attribute set is only valid if both reason and time limit are defined. The Batch API will throw an error if not all attributes are defined, so a check is not strictly necessary but it would be nice to know before starting a deploy that this will fail if both are not defined.