[meta] Cost efficiency

[eu9ene]

Let's make sure the infrastructure settings are adjusted based on the current needs of the pipeline. There shouldn't be any waste but at the same time infrastructure shouldn't limit experimentation speed. We can also pursue some quick optimizations here.

Optimization

Preview Give feedback

1. Adjust OpusTrainer settings in CI firefox-translations-training#455
   cost & perf +1
2. use smaller instances for PRs firefox-translations-training#419
   cost & perf +1
3. Full audit of resource utilization and adjustments firefox-translations-training#414
   cost & perf +1
4. Reduce the number of vCPUs for bicleaner machines firefox-translations-training#394
   cost & perf +1
5. Do not run bicleaner step if the threshold is 0 firefox-translations-training#415
   cost & perf +1
6. Investigate optimizing the CI training run firefox-translations-training#459
   cost & perf +1
7. Examine strategies for more efficient alignments firefox-translations-training#663
   cost & perf high resource +2
   
8. Investigate CTranslate2 for translating sentences with teacher model firefox-translations-training#165
   cost & perf +1
   
9. Process alignments in chunks firefox-translations-training#739
   cost & perf +1
10. Investigate removing teacher ensemble training #778
    cost & perf experiment +2
    
11. Improve GPU utilization for "translate" tasks firefox-translations-training#785
    cost & perf +1
12. Improve GPU utilization in student training firefox-translations-training#783
    cost & perf +1
13. Limit the amount of data used for distillation #905
    cost & perf +1

On-premises cluster

Preview Give feedback

1. allow for runtime selection of GPU workers firefox-translations-training#300
   enhancement on-prem taskcluster +3
2. Automatically adjust machine dependent settings firefox-translations-training#253
   on-prem taskcluster +2
3. Add support for on-prem workers firefox-translations-training#230
   cost & perf on-prem taskcluster +3
4. Consider switching to Docker for all tasks firefox-translations-training#391
   enhancement taskcluster +2

[gregtatum]

I don't know that I have a specific issue to tie this to, but here is a performance profile of all the en-ca training tasks: https://share.firefox.dev/49SotyG

[gregtatum]

And here is a visualization of the training run: https://gregtatum.github.io/taskcluster-tools/?taskGroupIds=GU9ZyWFhRDe_nxlAHcen8g&fetchDependentTasks=true&ignoredTaskGroupIds=aCUq1LgtQCeHkmk-wJhbtQ%2CcKY8O-guTcKXZXRYpWLGiw%2CdYEG7rEWS8esjEvDrzmtrA%2CXvaP5gPWSO68cWmub-moLw%2CO-49dfOARwumu49ZkmhhGQ