Make workers configurable

[eu9ene]

Currently, it requires changing each step if we say, want to train the whole pipeline on a different worker pool. For example, we might want to experiment with different GPU types or run the pipeline on the on-prem cluster and set the specific GPU we want to use.

Can we configure workers from the training config? I think it should be possible with transforms.

[gabrielBusta]

It sounds possible. The transform can load the config and tweak the worker field in the task

[bhearsum]

Note that transforms will work if each worker pool has a consistent value. If we have multiple workers in the same pool that require different values, we won't know which ones are correct until runtime.

IMO, in an ideal world, the workers would set GPUS/WORKSPACE themselves, and the tasks would inherit these values (since they are a property of the worker). I'm not sure at this point if that's possible. If not, runtime detection in a script is necessary for this case.

[eu9ene]

Yes, ideally those values should depend on a worker. Related to #253

[bhearsum]

We talked about this a bit on Zoom today. We agreed that in the short, and maybe medium, term that we would keep worker pools consistent as far as their type and number of GPUs go. With this in mind we should be able to define the GPUS and WORKSPACE values in https://github.com/mozilla/firefox-translations-training/blob/main/taskcluster/ci/config.yml (this will have to be in a new top level key, as https://github.com/taskcluster/taskgraph/blob/0401b911ec0a5d3a8b66bdacc183362aa7811871/src/taskgraph/config.py#L43 does not allow extra configuration). From there, a transform can pull the values and insert them into the env of the necessary tasks.

When we have other worker pools we want to train on, we'll just need to adjust those entries and the worker provisioner/worker-type in config.yml to switch to, eg: the on prem workers. So for example, right now much of the training is done on:

        b-linux-v100-gpu-4-1tb:
            provisioner: '{trust-domain}-{level}'
            implementation: generic-worker
            os: linux
            worker-type: '{alias}'

When the on prem machines are available, this entry would change to something like:

        b-linux-v100-gpu-4-1tb:
            provisioner: '{trust-domain}-onprem'
            implementation: generic-worker
            os: linux
            worker-type: snakepit

(The b-linux-v100-gpu-4-1tb key is what the kinds use to look up workers, while the provisioner and worker-type are what map it back to the machines doing the work.)