Skip to content

Commit

Permalink
Support fetches in dependencies_dummy
Browse files Browse the repository at this point in the history
The fix for mozilla#653 is going to involve dummy tasks that copy artifacts from their upstreams. To support this, we need our little helper transform to do a few things:
* Allow for one fewer upstream dependency, to make room for a `docker-image` task upstream
* Be able to add `fetches` entries for the tasks upstream of it
* Store any fetched artifact names in `attributes`, to allow tasks _downstream_ of a dummy to easily find all artifacts that the dummy will republish

I consider the last part fairly hacky, but couldn't come up with anything better.
  • Loading branch information
bhearsum committed Jun 19, 2024
1 parent 2b9e53e commit 736f7a5
Showing 1 changed file with 20 additions and 4 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -17,12 +17,24 @@
from taskgraph import MAX_DEPENDENCIES
from taskgraph.transforms.base import TransformSequence

# One less than taskgraph, because some dummy tasks may depend on:
# - decision task (taskgraph MAX_DEPENDENCIES already accounts for this)
# - a docker image task (taskgraph MAX_DEPENDENCIES does _not_ account for this)
# - N other upstream tasks
OUR_MAX_DEPENDENCIES = MAX_DEPENDENCIES - 1

transforms = TransformSequence()


def yield_job(orig_job, deps, count):
def yield_job(orig_job, deps, fetches, count):
job = copy.deepcopy(orig_job)
job["dependencies"] = deps
if fetches:
job["fetches"] = fetches
job["attributes"]["fetched_artifacts"] = []
for artifacts in fetches.values():
for a in artifacts:
job["attributes"]["fetched_artifacts"].append(a["artifact"])
job["name"] = "{}-{}".format(orig_job["name"], count)

return job
Expand All @@ -33,14 +45,18 @@ def add_dependencies(config, jobs):
for job in jobs:
count = 1
deps = {}
fetches = {}

for dep_label in sorted(job["dependencies"].keys()):
deps[dep_label] = dep_label
if len(deps) == MAX_DEPENDENCIES:
yield yield_job(job, deps, count)
if job.get("fetches", {}).get(dep_label):
fetches[dep_label] = job["fetches"][dep_label]
if len(deps) == OUR_MAX_DEPENDENCIES:
yield yield_job(job, deps, fetches, count)
deps = {}
fetches = {}
count += 1

if deps:
yield yield_job(job, deps, count)
yield yield_job(job, deps, fetches, count)
count += 1

0 comments on commit 736f7a5

Please sign in to comment.