You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Sometimes we run into OOM and it's hard to say from the logs that it's the case. It looks like a preemption of a spot instance. We should be able to easily identify that the task was terminated because the machine was out of memory.
The text was updated successfully, but these errors were encountered:
eu9ene
added
the
taskcluster
Issues related to the Taskcluster implementation of the training pipeline
label
May 3, 2024
eu9ene
changed the title
OOM looks like preemption
OOM looks like a preemption
May 3, 2024
I don't think there's anything we can do to make this better in this repo nor taskgraph. This is a worker issue that's been filed as taskcluster/taskcluster#6894
Sometimes we run into OOM and it's hard to say from the logs that it's the case. It looks like a preemption of a spot instance. We should be able to easily identify that the task was terminated because the machine was out of memory.
The text was updated successfully, but these errors were encountered: