Task has too many dependencies #653
Comments
eu9ene
added
bug
|
The only workaround we have for this is adding tasks between |
To this point, I'm just going to republish artifacts in the dummy tasks. I looked the bicleaner tasks on one of the large recent training runs, and the artifacts totaled to ~25GB at rest. That costs ~$.40/month to store in GCP, so even if we had 100 of those size runs in a year we're looking at ~$500/year to store them. We can revisit this decision at some point, but I don't think it's worth fussing with an alternate solution for finding artifacts of an indirect upstream at this time. |
The fix for mozilla#653 is going to involve dummy tasks that copy artifacts from their upstreams. To support this, we need our little helper transform to do a few things: * Allow for one fewer upstream dependency, to make room for a `docker-image` task upstream * Be able to add `fetches` entries for the tasks upstream of it * Store any fetched artifact names in `attributes`, to allow tasks _downstream_ of a dummy to easily find all artifacts that the dummy will republish I consider the last part fairly hacky, but couldn't come up with anything better.
The fix for mozilla#653 is going to involve dummy tasks that copy artifacts from their upstreams. To support this, we need our little helper transform to do a few things: * Allow for one fewer upstream dependency, to make room for a `docker-image` task upstream * Be able to add `fetches` entries for the tasks upstream of it * Store any fetched artifact names in `attributes`, to allow tasks _downstream_ of a dummy to easily find all artifacts that the dummy will republish I consider the last part fairly hacky, but couldn't come up with anything better.
The fix for mozilla#653 is going to involve dummy tasks that copy artifacts from their upstreams. To support this, we need our little helper transform to do a few things: * Allow for one fewer upstream dependency, to make room for a `docker-image` task upstream * Be able to add `fetches` entries for the tasks upstream of it * Store any fetched artifact names in `attributes`, to allow tasks _downstream_ of a dummy to easily find all artifacts that the dummy will republish I consider the last part fairly hacky, but couldn't come up with anything better.
|
Another solution for this could be to chunk dataset tasks together. If we managed to chunk them by size we could avoid increasing the end to end runtime as well. This might not be great for caching purposes, but I wanted to mention it here for completeness. |
The fix for mozilla#653 is going to involve dummy tasks that copy artifacts from their upstreams. To support this, we need our little helper transform to do a few things: * Allow for one fewer upstream dependency, to make room for a `docker-image` task upstream * Be able to add `fetches` entries for the tasks upstream of it * Store any fetched artifact names in `attributes`, to allow tasks _downstream_ of a dummy to easily find all artifacts that the dummy will republish I consider the last part fairly hacky, but couldn't come up with anything better.
|
It turns out that the current limit is likely not a hard limit these days. We're working an removing or greatly increasing this limit in Taskcluster in taskcluster/taskcluster#7151. |
|
The Firefox CI cluster now supports up to 10,000 dependencies 🥳. We'll still need a taskgraph change to have that allow for it. |
Most notably, this includes a change that allows us to have up to 10,000 dependencies per task, which will fix mozilla#653.
https://firefox-ci-tc.services.mozilla.com/tasks/ZqlokLMTQG-pZPJtK9UnOw/runs/0/logs/public/logs/live.log
Exception: task merge-corpus/merge-corpus-da-en has too many dependencies (105 > 99)The text was updated successfully, but these errors were encountered: