SUP-3258 - Implement .spec.job.activeDeadlineSeconds #497
Conversation
There was a problem hiding this comment.
LGTM
Other than one comment below, ideally this would also have a test. The easiest way currently is probably an integration test that uses a pipeline YAML fixture with a small activeDeadlineSeconds and a command that sleeps longer than it. But it may be tricky to take image pull time into account, so the deadline and sleep would probably be at least 30 seconds to avoid flakiness.
| @@ -75,6 +76,7 @@ func New(logger *zap.Logger, client kubernetes.Interface, cfg Config) *worker { | |||
| } | |||
|
|
|||
| type KubernetesPlugin struct { | |||
| Job *batchv1.Job `json:"job,omitempty"` | |||
There was a problem hiding this comment.
Using *batchv1.Job here could get confusing - someone might try to write:
plugins:
- kubernetes:
job:
template:
spec:
containers:
...which wouldn't be rejected for any failure to parse, and the user might expect that defining containers that way would work but instead it would be silently ignored.
(Maybe one day the whole KubernetesPlugin should be changed to revolve around a *batchv1.Job instead of defining pieces of it. Needs thought.)
In the meantime I would suggest either just adding JobActiveDeadlineSeconds as a KubernetesPlugin field directly, or maybe wrapping it in a new Job struct that can be grown over time to include other job parameters.
There was a problem hiding this comment.
This is step 1 of a long journey to fully transition to *batchv1.Job. I went back and forth on whether to implement it this way and possibly cause confusion for users that try to leverage the entire Job spec. There are additional tickets we have that will continue to build off of the Job spec (backoffLimit, podFailurePolicy, etc.) bit by bit. We could implement this (and the future additional fields) as fields in the plugin and eventually bring them back into the full Job spec when the time comes to more precisely control the feature set.
petetomasik
force-pushed
the
SUP-3258-job-active-deadline-seconds
branch
from
fd05068 to
7a13241
Compare
Implements
.spec.job.activeDeadlineSecondsas a means of setting runtime boundaries for Kubernetes jobs that are unable to complete within a "reasonable" amount of time.This is configured with a default value of
21600seconds (6 hours) in the controller configuration. This value can be overridden by the user via the command line--job-active-deadline-seconds 43200, directly in the controller configurationjob-active-deadline-seconds: 43200, or on a per Buildkite job basis via thekubernetesplugin:At this time
.spec.job.activeDeadlineSecondswill not be able to be disabled, as we agree that a job runtime boundary should be defined at all times.Fixes #480
Fixes #479
Fixes #463