-
-
Notifications
You must be signed in to change notification settings - Fork 519
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
.github/workflows/docker.yml: Interrupt the build before the 6 hour cancellation #36670
Conversation
Is this for workflow that is canceled by timeout? Does this allow to investigate the image (with |
Yes, the main use case is for debugging these timeouts. But I could also imagine using it to get earlier access to the image of a failing build. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK.
Thanks... but it wasn't ready for review; it doesn't work yet |
Ah, sorry.
The code doesn't work? Or do you plan to do more? |
This approach works locally as described, but in GH Actions the runner dies too quickly. I'll have to make changes and experiments |
Isn't |
It's possible that I may be making a stupid mistake in setting up the trap in this step. I cannot see any sign of it being triggered. Help with this is very welcome! |
d14e820
to
e165caf
Compare
e165caf
to
a846e18
Compare
I've switched to a simpler approach, suitable for the main use case. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK, if it works for you.
Thanks! |
883e05f
to
e349b00
Compare
a846e18
to
7df3b63
Compare
Rebased onto the proper rc2. |
Setting to blocker because there are more build hangs, possibly from contourpy, in https://github.com/sagemath/sage/actions/runs/6842945144/job/18611623959#step:11:1932, https://github.com/sagemath/sage/actions/runs/6842945144/job/18611622942#step:11:2109, https://github.com/sagemath/sage/actions/runs/6842945144/job/18611624721 |
Documentation preview for this PR (built with commit 7df3b63; changes) is ready! 🎉 |
Follow-up after: - sagemath#36670 This is working well, for example see https://github.com/mkoeppe/sage/ac tions/runs/6874334523/job/18695809308#step:11:5345, making it possible to obtain the image of the hanging build - https://github.com/mkoeppe/sa ge/actions/runs/6874334523/job/18695809308#step:15:25 However, `pkill` is not present on many of our `-minimal` configurations (which test our documented minimal build prerequisites). For example `debian-sid-minimal`, as seen in https://github.com/mkoeppe/sage/actions /runs/6874334523/job/18695811066#step:11:2262, rendering the mechanism ineffective there. (Neither are `ps` and `killall`.) Examples: - `docker run -it ghcr.io/sagemath/sage/sage-debian-bookworm-minimal- with-system-packages:dev bash -c "command -v pkill" ` - `docker run -it ghcr.io/sagemath/sage/sage-fedora-38-minimal-with- system-packages:dev bash -c "command -v pkill"` - whereas `docker run -it ghcr.io/sagemath/sage/sage-fedora-38-standard- with-system-packages:dev bash -c "command -v pkill" ` --> `/usr/bin/pkill`. We replace `pkill` by more primitive means. Test run (with timeout set to ~11 minutes for testing purposes): https:/ /github.com/mkoeppe/sage/actions/runs/6883232154/job/18723359505#step:15 :25 <!-- ^^^^^ Please provide a concise, informative and self-explanatory title. Don't put issue numbers in there, do this in the PR body below. For example, instead of "Fixes sagemath#1234" use "Introduce new method to calculate 1+1" --> <!-- Describe your changes here in detail --> <!-- Why is this change required? What problem does it solve? --> <!-- If this PR resolves an open issue, please link to it here. For example "Fixes sagemath#12345". --> <!-- If your change requires a documentation PR, please link it appropriately. --> ### 📝 Checklist <!-- Put an `x` in all the boxes that apply. --> <!-- If your change requires a documentation PR, please link it appropriately --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> <!-- Feel free to remove irrelevant items. --> - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies <!-- List all open PRs that this PR logically depends on - sagemath#12345: short description why this is a dependency - sagemath#34567: ... --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> URL: sagemath#36726 Reported by: Matthias Köppe Reviewer(s): Kwankyu Lee, Nathan Dunfield
Follow-up after: - sagemath#36670 This is working well, for example see https://github.com/mkoeppe/sage/ac tions/runs/6874334523/job/18695809308#step:11:5345, making it possible to obtain the image of the hanging build - https://github.com/mkoeppe/sa ge/actions/runs/6874334523/job/18695809308#step:15:25 However, `pkill` is not present on many of our `-minimal` configurations (which test our documented minimal build prerequisites). For example `debian-sid-minimal`, as seen in https://github.com/mkoeppe/sage/actions /runs/6874334523/job/18695811066#step:11:2262, rendering the mechanism ineffective there. (Neither are `ps` and `killall`.) Examples: - `docker run -it ghcr.io/sagemath/sage/sage-debian-bookworm-minimal- with-system-packages:dev bash -c "command -v pkill" ` - `docker run -it ghcr.io/sagemath/sage/sage-fedora-38-minimal-with- system-packages:dev bash -c "command -v pkill"` - whereas `docker run -it ghcr.io/sagemath/sage/sage-fedora-38-standard- with-system-packages:dev bash -c "command -v pkill" ` --> `/usr/bin/pkill`. We replace `pkill` by more primitive means. Test run (with timeout set to ~11 minutes for testing purposes): https:/ /github.com/mkoeppe/sage/actions/runs/6883232154/job/18723359505#step:15 :25 <!-- ^^^^^ Please provide a concise, informative and self-explanatory title. Don't put issue numbers in there, do this in the PR body below. For example, instead of "Fixes sagemath#1234" use "Introduce new method to calculate 1+1" --> <!-- Describe your changes here in detail --> <!-- Why is this change required? What problem does it solve? --> <!-- If this PR resolves an open issue, please link to it here. For example "Fixes sagemath#12345". --> <!-- If your change requires a documentation PR, please link it appropriately. --> ### 📝 Checklist <!-- Put an `x` in all the boxes that apply. --> <!-- If your change requires a documentation PR, please link it appropriately --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> <!-- Feel free to remove irrelevant items. --> - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies <!-- List all open PRs that this PR logically depends on - sagemath#12345: short description why this is a dependency - sagemath#34567: ... --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> URL: sagemath#36726 Reported by: Matthias Köppe Reviewer(s): Kwankyu Lee, Nathan Dunfield, William Stein
Follow-up after: - sagemath#36670 This is working well, for example see https://github.com/mkoeppe/sage/ac tions/runs/6874334523/job/18695809308#step:11:5345, making it possible to obtain the image of the hanging build - https://github.com/mkoeppe/sa ge/actions/runs/6874334523/job/18695809308#step:15:25 However, `pkill` is not present on many of our `-minimal` configurations (which test our documented minimal build prerequisites). For example `debian-sid-minimal`, as seen in https://github.com/mkoeppe/sage/actions /runs/6874334523/job/18695811066#step:11:2262, rendering the mechanism ineffective there. (Neither are `ps` and `killall`.) Examples: - `docker run -it ghcr.io/sagemath/sage/sage-debian-bookworm-minimal- with-system-packages:dev bash -c "command -v pkill" ` - `docker run -it ghcr.io/sagemath/sage/sage-fedora-38-minimal-with- system-packages:dev bash -c "command -v pkill"` - whereas `docker run -it ghcr.io/sagemath/sage/sage-fedora-38-standard- with-system-packages:dev bash -c "command -v pkill" ` --> `/usr/bin/pkill`. We replace `pkill` by more primitive means. Test run (with timeout set to ~11 minutes for testing purposes): https:/ /github.com/mkoeppe/sage/actions/runs/6883232154/job/18723359505#step:15 :25 <!-- ^^^^^ Please provide a concise, informative and self-explanatory title. Don't put issue numbers in there, do this in the PR body below. For example, instead of "Fixes sagemath#1234" use "Introduce new method to calculate 1+1" --> <!-- Describe your changes here in detail --> <!-- Why is this change required? What problem does it solve? --> <!-- If this PR resolves an open issue, please link to it here. For example "Fixes sagemath#12345". --> <!-- If your change requires a documentation PR, please link it appropriately. --> ### 📝 Checklist <!-- Put an `x` in all the boxes that apply. --> <!-- If your change requires a documentation PR, please link it appropriately --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> <!-- Feel free to remove irrelevant items. --> - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies <!-- List all open PRs that this PR logically depends on - sagemath#12345: short description why this is a dependency - sagemath#34567: ... --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> URL: sagemath#36726 Reported by: Matthias Köppe Reviewer(s): Kwankyu Lee, Nathan Dunfield, William Stein
Follow-up after: - sagemath#36670 This is working well, for example see https://github.com/mkoeppe/sage/ac tions/runs/6874334523/job/18695809308#step:11:5345, making it possible to obtain the image of the hanging build - https://github.com/mkoeppe/sa ge/actions/runs/6874334523/job/18695809308#step:15:25 However, `pkill` is not present on many of our `-minimal` configurations (which test our documented minimal build prerequisites). For example `debian-sid-minimal`, as seen in https://github.com/mkoeppe/sage/actions /runs/6874334523/job/18695811066#step:11:2262, rendering the mechanism ineffective there. (Neither are `ps` and `killall`.) Examples: - `docker run -it ghcr.io/sagemath/sage/sage-debian-bookworm-minimal- with-system-packages:dev bash -c "command -v pkill" ` - `docker run -it ghcr.io/sagemath/sage/sage-fedora-38-minimal-with- system-packages:dev bash -c "command -v pkill"` - whereas `docker run -it ghcr.io/sagemath/sage/sage-fedora-38-standard- with-system-packages:dev bash -c "command -v pkill" ` --> `/usr/bin/pkill`. We replace `pkill` by more primitive means. Test run (with timeout set to ~11 minutes for testing purposes): https:/ /github.com/mkoeppe/sage/actions/runs/6883232154/job/18723359505#step:15 :25 <!-- ^^^^^ Please provide a concise, informative and self-explanatory title. Don't put issue numbers in there, do this in the PR body below. For example, instead of "Fixes sagemath#1234" use "Introduce new method to calculate 1+1" --> <!-- Describe your changes here in detail --> <!-- Why is this change required? What problem does it solve? --> <!-- If this PR resolves an open issue, please link to it here. For example "Fixes sagemath#12345". --> <!-- If your change requires a documentation PR, please link it appropriately. --> ### 📝 Checklist <!-- Put an `x` in all the boxes that apply. --> <!-- If your change requires a documentation PR, please link it appropriately --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> <!-- Feel free to remove irrelevant items. --> - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies <!-- List all open PRs that this PR logically depends on - sagemath#12345: short description why this is a dependency - sagemath#34567: ... --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> URL: sagemath#36726 Reported by: Matthias Köppe Reviewer(s): Kwankyu Lee, Nathan Dunfield, William Stein
Follow-up after: - sagemath#36670 This is working well, for example see https://github.com/mkoeppe/sage/ac tions/runs/6874334523/job/18695809308#step:11:5345, making it possible to obtain the image of the hanging build - https://github.com/mkoeppe/sa ge/actions/runs/6874334523/job/18695809308#step:15:25 However, `pkill` is not present on many of our `-minimal` configurations (which test our documented minimal build prerequisites). For example `debian-sid-minimal`, as seen in https://github.com/mkoeppe/sage/actions /runs/6874334523/job/18695811066#step:11:2262, rendering the mechanism ineffective there. (Neither are `ps` and `killall`.) Examples: - `docker run -it ghcr.io/sagemath/sage/sage-debian-bookworm-minimal- with-system-packages:dev bash -c "command -v pkill" ` - `docker run -it ghcr.io/sagemath/sage/sage-fedora-38-minimal-with- system-packages:dev bash -c "command -v pkill"` - whereas `docker run -it ghcr.io/sagemath/sage/sage-fedora-38-standard- with-system-packages:dev bash -c "command -v pkill" ` --> `/usr/bin/pkill`. We replace `pkill` by more primitive means. Test run (with timeout set to ~11 minutes for testing purposes): https:/ /github.com/mkoeppe/sage/actions/runs/6883232154/job/18723359505#step:15 :25 <!-- ^^^^^ Please provide a concise, informative and self-explanatory title. Don't put issue numbers in there, do this in the PR body below. For example, instead of "Fixes sagemath#1234" use "Introduce new method to calculate 1+1" --> <!-- Describe your changes here in detail --> <!-- Why is this change required? What problem does it solve? --> <!-- If this PR resolves an open issue, please link to it here. For example "Fixes sagemath#12345". --> <!-- If your change requires a documentation PR, please link it appropriately. --> ### 📝 Checklist <!-- Put an `x` in all the boxes that apply. --> <!-- If your change requires a documentation PR, please link it appropriately --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> <!-- Feel free to remove irrelevant items. --> - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies <!-- List all open PRs that this PR logically depends on - sagemath#12345: short description why this is a dependency - sagemath#34567: ... --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> URL: sagemath#36726 Reported by: Matthias Köppe Reviewer(s): Kwankyu Lee, Nathan Dunfield, William Stein
Follow-up after: - sagemath#36670 This is working well, for example see https://github.com/mkoeppe/sage/ac tions/runs/6874334523/job/18695809308#step:11:5345, making it possible to obtain the image of the hanging build - https://github.com/mkoeppe/sa ge/actions/runs/6874334523/job/18695809308#step:15:25 However, `pkill` is not present on many of our `-minimal` configurations (which test our documented minimal build prerequisites). For example `debian-sid-minimal`, as seen in https://github.com/mkoeppe/sage/actions /runs/6874334523/job/18695811066#step:11:2262, rendering the mechanism ineffective there. (Neither are `ps` and `killall`.) Examples: - `docker run -it ghcr.io/sagemath/sage/sage-debian-bookworm-minimal- with-system-packages:dev bash -c "command -v pkill" ` - `docker run -it ghcr.io/sagemath/sage/sage-fedora-38-minimal-with- system-packages:dev bash -c "command -v pkill"` - whereas `docker run -it ghcr.io/sagemath/sage/sage-fedora-38-standard- with-system-packages:dev bash -c "command -v pkill" ` --> `/usr/bin/pkill`. We replace `pkill` by more primitive means. Test run (with timeout set to ~11 minutes for testing purposes): https:/ /github.com/mkoeppe/sage/actions/runs/6883232154/job/18723359505#step:15 :25 <!-- ^^^^^ Please provide a concise, informative and self-explanatory title. Don't put issue numbers in there, do this in the PR body below. For example, instead of "Fixes sagemath#1234" use "Introduce new method to calculate 1+1" --> <!-- Describe your changes here in detail --> <!-- Why is this change required? What problem does it solve? --> <!-- If this PR resolves an open issue, please link to it here. For example "Fixes sagemath#12345". --> <!-- If your change requires a documentation PR, please link it appropriately. --> ### 📝 Checklist <!-- Put an `x` in all the boxes that apply. --> <!-- If your change requires a documentation PR, please link it appropriately --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> <!-- Feel free to remove irrelevant items. --> - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies <!-- List all open PRs that this PR logically depends on - sagemath#12345: short description why this is a dependency - sagemath#34567: ... --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> URL: sagemath#36726 Reported by: Matthias Köppe Reviewer(s): Kwankyu Lee, Nathan Dunfield, William Stein
Follow-up after: - sagemath#36670 This is working well, for example see https://github.com/mkoeppe/sage/ac tions/runs/6874334523/job/18695809308#step:11:5345, making it possible to obtain the image of the hanging build - https://github.com/mkoeppe/sa ge/actions/runs/6874334523/job/18695809308#step:15:25 However, `pkill` is not present on many of our `-minimal` configurations (which test our documented minimal build prerequisites). For example `debian-sid-minimal`, as seen in https://github.com/mkoeppe/sage/actions /runs/6874334523/job/18695811066#step:11:2262, rendering the mechanism ineffective there. (Neither are `ps` and `killall`.) Examples: - `docker run -it ghcr.io/sagemath/sage/sage-debian-bookworm-minimal- with-system-packages:dev bash -c "command -v pkill" ` - `docker run -it ghcr.io/sagemath/sage/sage-fedora-38-minimal-with- system-packages:dev bash -c "command -v pkill"` - whereas `docker run -it ghcr.io/sagemath/sage/sage-fedora-38-standard- with-system-packages:dev bash -c "command -v pkill" ` --> `/usr/bin/pkill`. We replace `pkill` by more primitive means. Test run (with timeout set to ~11 minutes for testing purposes): https:/ /github.com/mkoeppe/sage/actions/runs/6883232154/job/18723359505#step:15 :25 <!-- ^^^^^ Please provide a concise, informative and self-explanatory title. Don't put issue numbers in there, do this in the PR body below. For example, instead of "Fixes sagemath#1234" use "Introduce new method to calculate 1+1" --> <!-- Describe your changes here in detail --> <!-- Why is this change required? What problem does it solve? --> <!-- If this PR resolves an open issue, please link to it here. For example "Fixes sagemath#12345". --> <!-- If your change requires a documentation PR, please link it appropriately. --> ### 📝 Checklist <!-- Put an `x` in all the boxes that apply. --> <!-- If your change requires a documentation PR, please link it appropriately --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> <!-- Feel free to remove irrelevant items. --> - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies <!-- List all open PRs that this PR logically depends on - sagemath#12345: short description why this is a dependency - sagemath#34567: ... --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> URL: sagemath#36726 Reported by: Matthias Köppe Reviewer(s): Kwankyu Lee, Nathan Dunfield, William Stein
Follow-up after: - sagemath#36670 This is working well, for example see https://github.com/mkoeppe/sage/ac tions/runs/6874334523/job/18695809308#step:11:5345, making it possible to obtain the image of the hanging build - https://github.com/mkoeppe/sa ge/actions/runs/6874334523/job/18695809308#step:15:25 However, `pkill` is not present on many of our `-minimal` configurations (which test our documented minimal build prerequisites). For example `debian-sid-minimal`, as seen in https://github.com/mkoeppe/sage/actions /runs/6874334523/job/18695811066#step:11:2262, rendering the mechanism ineffective there. (Neither are `ps` and `killall`.) Examples: - `docker run -it ghcr.io/sagemath/sage/sage-debian-bookworm-minimal- with-system-packages:dev bash -c "command -v pkill" ` - `docker run -it ghcr.io/sagemath/sage/sage-fedora-38-minimal-with- system-packages:dev bash -c "command -v pkill"` - whereas `docker run -it ghcr.io/sagemath/sage/sage-fedora-38-standard- with-system-packages:dev bash -c "command -v pkill" ` --> `/usr/bin/pkill`. We replace `pkill` by more primitive means. Test run (with timeout set to ~11 minutes for testing purposes): https:/ /github.com/mkoeppe/sage/actions/runs/6883232154/job/18723359505#step:15 :25 <!-- ^^^^^ Please provide a concise, informative and self-explanatory title. Don't put issue numbers in there, do this in the PR body below. For example, instead of "Fixes sagemath#1234" use "Introduce new method to calculate 1+1" --> <!-- Describe your changes here in detail --> <!-- Why is this change required? What problem does it solve? --> <!-- If this PR resolves an open issue, please link to it here. For example "Fixes sagemath#12345". --> <!-- If your change requires a documentation PR, please link it appropriately. --> ### 📝 Checklist <!-- Put an `x` in all the boxes that apply. --> <!-- If your change requires a documentation PR, please link it appropriately --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> <!-- Feel free to remove irrelevant items. --> - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies <!-- List all open PRs that this PR logically depends on - sagemath#12345: short description why this is a dependency - sagemath#34567: ... --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> URL: sagemath#36726 Reported by: Matthias Köppe Reviewer(s): Kwankyu Lee, Nathan Dunfield, William Stein
When a CI Linux workflow is canceled, because we are using
docker build
, no image is written out.Here we add an input parameter
timeout
, defaulting to 5.5 hours, after which we kill themake
process inside of the build container.Then our script (in tox) writes out and pushes an image (marked as "-failed") to ghcr.io, enabling better diagnostics.
See https://github.com/mkoeppe/sage/actions/runs/6831246509/job/18580446041 for a test run where for demonstration purposes I set the timeout to 6 minutes.
📝 Checklist
⌛ Dependencies