Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Investigate optimizing the CI training run #459

Open
Tracked by #453
gregtatum opened this issue Feb 21, 2024 · 2 comments
Open
Tracked by #453

Investigate optimizing the CI training run #459

gregtatum opened this issue Feb 21, 2024 · 2 comments
Labels
cost & perf Speeding up and lowering cost for the pipeline

Comments

@gregtatum
Copy link
Member

It would be nice to optimize the end to end time. Here it is 1 hour and 25 minutes
https://share.firefox.dev/3I192Z3

The training steps are the ones that take the longest time. We could try using less data, fewer epochs of training, and smaller model sizes.

Step Time
Teacher 12:55
Student 12:43
Finetune Student 13:41

@eu9ene you mentioned something about quantization failures in a meeting when we discussed this, can you elaborate on that?
@gregtatum gregtatum added the cost & perf Speeding up and lowering cost for the pipeline label Feb 21, 2024
@gregtatum gregtatum mentioned this issue Feb 21, 2024
Open
@eu9ene
Copy link
Collaborator

eu9ene commented Feb 26, 2024

It's basically this: #455

you mentioned something about quantization failures in a meeting when we discussed this, can you elaborate on that?

I'm not sure, we should investigate

@eu9ene
Copy link
Collaborator

eu9ene commented May 22, 2024

Something happens on [taskcluster-proxy] Successfully refreshed taskcluster-proxy credentials:. I see two 20 minutes gaps when this line appears: https://firefox-ci-tc.services.mozilla.com/tasks/TnmPTMeqSPWB727etTAAaw/runs/1/logs/live/public/logs/live.log

nordzilla pushed a commit that referenced this issue Sep 19, 2024
@XapaJIaMnu @jelmervdl
Remove wormhole references (#459)
534ed37 
* Remove warmhole references

* Remove more references to the WORMHOLE

* Update marian to wormhole removed marian

* Whoops

---------

Co-authored-by: Jelmer van der Linde <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
cost & perf Speeding up and lowering cost for the pipeline
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@gregtatum @eu9ene