Fix learning rate gap on resume

[Laughing-q]

🛠️ PR Summary

_{Made with ❤️ by Ultralytics Actions}

🌟 Summary

Improvements in training scheduling and more informative logging.

📊 Key Changes

• Optimizer step warning suppression moved and simplified within the training loop.
• Scheduling based on time now occurs directly after an epoch ends rather than inside a warning suppression block.
• Introduction of a more descriptive logging prefix (EarlyStopping: ) for better clarity on training halts due to lack of improvement.

🎯 Purpose & Impact

• Enhanced Efficiency 🏃‍♂️: Moving the learning rate scheduler step outside of a warning suppression block streamlines the training process, potentially reducing confusion and improving code maintainability.
• Time-Based Scheduling 🕒: Adjusting the scheduler after each epoch based on the average epoch duration and desired training time allows for more precise control over the training duration, making it easier to fit training into specific time slots.
• Improved User Experience 📈: Introducing a colored "EarlyStopping:" prefix to logs makes it immediately clear when training has been halted due to no observed improvement, enhancing user understanding and control over the training process.

These changes aim to simplify the training loop for better performance, more precise time management, and clearer communication of important training events, all of which contribute to a more efficient and user-friendly training experience.

[codecov]

Codecov Report

Attention: Patch coverage is 45.45455% with 6 lines in your changes are missing coverage. Please review.

Project coverage is 76.67%. Comparing base (e5f4f5c) to head (02a652e).

❗ Current head 02a652e differs from pull request most recent head e8cfdd6. Consider uploading reports for the commit e8cfdd6 to get more accurate results

Files	Patch %	Lines
ultralytics/engine/trainer.py	44.44%	5 Missing ⚠️
ultralytics/utils/torch_utils.py	50.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #9468       +/-   ##
===========================================
+ Coverage   37.99%   76.67%   +38.67%     
===========================================
  Files         121      120        -1     
  Lines       15277    15175      -102     
===========================================
+ Hits         5805    11635     +5830     
+ Misses       9472     3540     -5932     

Flag	Coverage Δ
Benchmarks	36.30% <9.09%> (?)
GPU	38.23% <45.45%> (+0.23%)	⬆️
Tests	71.93% <45.45%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[Laughing-q]

@glenn-jocher this PR fixed the lr issue when resuming. I tested a training locally with interrupted at epoch 5/10 then resume the training.

Also I tested time training by:

yolo detect train data=runs/data/coco.yaml time=0.01
yolo detect train data=runs/data/coco.yaml time=0.02
yolo detect train data=runs/data/coco.yaml time=0.05

For quick verification, I manually set the train set to coco/val2017 which contains 5000 images. And all these three training behave exactly the same before and after this fix.

• yolo detect train data=runs/data/coco.yaml time=0.01 finished the training at the middle of the first epoch.
• yolo detect train data=runs/data/coco.yaml time=0.02 finished the training right after 1 epoch.
• yolo detect train data=runs/data/coco.yaml time=0.05 finished the training right after 2 epoch.

[Laughing-q]

@glenn-jocher please take a look and check if I break anything. Thanks!

[glenn-jocher]

@Laughing-q that would be super nice we could get rid of all that timed training logic, but I think I had to include it for a reason. One test for timed training is we need to run with less time and more time than the default epochs count. I'll run some tests on this.

[glenn-jocher]

Oh I think I remember now, we need to update the LR scheduler to hit lrf on epochs epoch, I think this logic handles that as epochs keeps updating during the training.

If we don't update the LR scheduler then timed training won't be optimal as the LR scheduler will be fixed to hit lrf on the default epochs instead of the updated epochs.

So timed training does 3 things at the end of each epoch:

1. Check the time that's been spent on the past epochs and figure out an average time per epoch.
2. Compute new epochs to end training perfectly on the requested time.
3. Update the LR scheduler to hit final lr on the new final epochs.

[glenn-jocher]

@Laughing-q ok I moved the scheduler.step() line to the beginning of the train loop instead of the end. The new LR will always be 1 step ahead of the previous one, but I think this fine. Resume looks good now. I CTRL+C a run 3 times and resumed 3 times here:

What do you think?

[Laughing-q]

@glenn-jocher Looks good to me!
Actually this is interesting, moving scheduler.step() to the beginning seems to be the correct one. I launched four training to test the lr change:

• main branch, default settings vs warmup_epochs=0(to check out what the lr looks like if scheduler starts working at the beginning.)
  

• current branch, default settings vs warmup_epochs=0, which does scheduler.step() at the beginning.
  

[glenn-jocher]

Nice!!

[glenn-jocher]

@Laughing-q PR merged!