Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Build should time out if it takes more than X hours #195

Closed
jlebon opened this issue Feb 6, 2020 · 3 comments · Fixed by #214
Closed

Build should time out if it takes more than X hours #195

jlebon opened this issue Feb 6, 2020 · 3 comments · Fixed by #214

Comments

@jlebon
Copy link
Member

jlebon commented Feb 6, 2020

E.g. we had a build recently just hang on S3 uploads.

Or could try to be fancier and do different timeouts per stage. But a big wrapping timeout would work to begin with.
@jlebon jlebon changed the title Build should time out if build takes more than X hours Build should time out if it takes more than X hours Feb 6, 2020
@dustymabe
Copy link
Member

👍

@dustymabe
Copy link
Member

hit this again.. I just killed https://jenkins-fedora-coreos.apps.ci.centos.org/job/fedora-coreos/job/fedora-coreos-fedora-coreos-pipeline/11960/ which has been running almost a day.

jlebon added a commit to jlebon/fedora-coreos-pipeline that referenced this issue Mar 25, 2020
@jlebon
pipeline: add overall timeout of 3h
9baac33 
Let's cap build runs to 3h. Builds should never take that long. Ideally,
we'd do timeouts per stage, and that could make sense (e.g. signing for
example could in the worst case take more than 1h, but building the
qcow2 shouldn't really take more than 15m), though just having one
overall timeout to start with will at least prevent jobs from getting
stuck anywhere in the process forever and taking up a pod in a cluster.

Closes: coreos#195
@jlebon jlebon mentioned this issue Mar 25, 2020
Merged
@jlebon
Copy link
Member Author

jlebon commented Mar 25, 2020

#214

jlebon added a commit to jlebon/fedora-coreos-pipeline that referenced this issue Mar 26, 2020
@jlebon
pipeline: add overall timeout of 4h
2a87dd4 
Let's cap build runs to 4h. Builds should never take that long. Ideally,
we'd do timeouts per stage, and that could make sense (e.g. signing for
example could in the worst case take more than 1h, but building the
qcow2 shouldn't really take more than 15m), though just having one
overall timeout to start with will at least prevent jobs from getting
stuck anywhere in the process forever and taking up a pod in a cluster.

Closes: coreos#195
dustymabe pushed a commit that referenced this issue Mar 28, 2020
@jlebon @dustymabe
pipeline: add overall timeout of 4h
093801e 
Let's cap build runs to 4h. Builds should never take that long. Ideally,
we'd do timeouts per stage, and that could make sense (e.g. signing for
example could in the worst case take more than 1h, but building the
qcow2 shouldn't really take more than 15m), though just having one
overall timeout to start with will at least prevent jobs from getting
stuck anywhere in the process forever and taking up a pod in a cluster.

Closes: #195
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

pipeline: add overall timeout of 4h jlebon/fedora-coreos-pipeline
2 participants
@dustymabe @jlebon