[meta] Make the pipeline reliable enough to train many languages

[gregtatum]

This is the lists of tasks that we need to handle in order to ramp up our ability to train many languages. These are things that break training runs, make things difficult to use the pipeline, or make it difficult for multiple people to train at once.

Pipeline Usability

Preview Give feedback

1. switch to generic worker for all tasks firefox-translations-training#710
   taskcluster +1
2. Interactive task's shell disconnects periodically firefox-translations-training#355
   taskcluster +1
   
3. "Cancel all" action doesn't work firefox-translations-training#250
   taskcluster +1
   
4. Disable spot instances for training firefox-translations-training#356
   taskcluster +1
   
5. Add an ability to run starting from a specific task firefox-translations-training#227
   taskcluster tc-p1 +2
   
6. NetworkError while fetching log firefox-translations-training#200
   blocker bug taskcluster tc-p1 +4
   
7. Update worker-runner in translations generic worker image
8. Provide basic toolset in interactive tasks firefox-translations-training#330
   taskcluster tc-p1 +2
   
9. Permissions issues in interactive tasks firefox-translations-training#354
   taskcluster +1
   
10. Show config validation error firefox-translations-training#155
    enhancement taskcluster +2
    
11. Observability of resource utilization firefox-translations-training#182
    cost & perf taskcluster tc-p1 +3
    
12. When using a pre-trained model there shouldn't be dependencies on dataset steps firefox-translations-training#395
    taskcluster tc-p1 +2
    
13. [taskcluster:error] Error uploading "public/build/tmp/aln.fwd" artifact. ext.certificate.expiry < now firefox-translations-training#582
    taskcluster +1
14. Task has too many dependencies firefox-translations-training#653
    bug taskcluster +2
    
15. docker tasks on generic worker sometimes hit issues with caches firefox-translations-training#538
    taskcluster +1
    
16. Taskcluster allows max 100 datasets firefox-translations-training#655
    taskcluster +1
17. Some mtdata datasets fail because of long name firefox-translations-training#654
    taskcluster +1
18. Many random errors on CPU generic workers firefox-translations-training#630
    bug taskcluster +2
    
19. Evaluation fails on a pre-trained backward model firefox-translations-training#628
    bug taskcluster +2
    
20. clean corpus unable to fork sometimes on generic-worker/d2g firefox-translations-training#640
    taskcluster +1
21. collect_mono failed with Read error (39) : premature end firefox-translations-training#680
    taskcluster +1
    
22. OOM looks like a preemption firefox-translations-training#562
    taskcluster +1
23. Can't restart the pipeline to run distillation from where it left firefox-translations-training#711
    taskcluster +1
24. start_stage often reruns amost all "evaluate" tasks firefox-translations-training#728
    cost & perf taskcluster +2
25. [taskcluster:error] Error uploading artifact: S3 returned status code 400 which could be an intermittent issue firefox-translations-training#806
    bug taskcluster +2
26. CI run is slowed down by failing to download fetches firefox-translations-training#855
    bug cost & perf taskcluster +3

Bugs in Training

Preview Give feedback

1. Unstable training with OpusTrainer firefox-translations-training#314
   bug quality +2
   
2. OpusTrainer should disable early stopping until the final stage firefox-translations-training#293
   bug quality +2
   
3. opuscleaner fails to pip install firefox-translations-training#292
   blocker bug +2
4. Distillation is broken firefox-translations-training#272
   bug quality +2
   
5. Examine strategies for more efficient alignments firefox-translations-training#663
   cost & perf high resource +2
   
6. OpusCleaner supports only a limited set of languages firefox-translations-training#649
   language-coverage +1
7. Marian error in translate-corpus firefox-translations-training#679
   bug +1
8. bicleaner-ai-classify intermittently fails to download fasttext model firefox-translations-training#631
   bug taskcluster +2

[bhearsum]

Seeing as we're training many languages at once, should we call this done? #250 is the only remaining issue open in the list, and it is mostly fixed (aside from some UI jank in Taskcluster).

[eu9ene]

I think we're still struggling with a bunch of issues, so I'd add them here and keep this one open until we can confidently train new languages without breakage.

[bhearsum]

Between this and #490 there are a fairly hefty number of issues related to Taskcluster with various dependencies between them. I created a simple diagram with dependencies and current status to hopefully help with this, which I will try to keep up to date.