Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Can ExpDecay of learning rate start at some intermediate step? #1815

Closed
cossio opened this issue Dec 24, 2021 · 2 comments · Fixed by #1816
Closed

Can ExpDecay of learning rate start at some intermediate step? #1815

cossio opened this issue Dec 24, 2021 · 2 comments · Fixed by #1816

Comments

@cossio
Copy link
Contributor

cossio commented Dec 24, 2021

I would like to start training a model with a constant learning rate (LR) first, and then after some epochs, start decaying exponentially the LR. I think this is not an uncommon practice.

It would be nice to add this option to ExpDecay.
@cossio cossio mentioned this issue Dec 24, 2021
Merged
4 tasks
@ToucheSir
Copy link
Member

This is exactly the kind of thing a library like ParameterSchedulers.jl was designed to handle. See some of the surrounding discussion at https://github.com/FluxML/Flux.jl/search?q=scheduler&type=issues.

@CarloLucibello CarloLucibello mentioned this issue Dec 25, 2021
Closed
@CarloLucibello
Copy link
Member

Using ParameterSchedulers.jl this behavior can be obtained with a Sequence scheduler https://darsnack.github.io/ParameterSchedulers.jl/dev/docs/tutorials/complex-schedules.html

That said, a keyword argument is more convenient for these kinds of common cases. Since the PR is already there and it is very simple I'm ok with having it.

bors bot added a commit that referenced this issue Dec 28, 2021
@bors
Merge #1816
d8e31f3 
1816: ExpDecay start step r=DhairyaLGandhi a=cossio

Adds an option to `ExpDecay` which specifies the step at which the exponential decay of the learning rate starts.

Fixes #1815.

### PR Checklist

- [x] Tests are added
- [ ] Entry in NEWS.md
- [x] Documentation, if applicable
- [ ] API changes require approval from a committer (different from the author, if applicable)


Co-authored-by: cossio <[email protected]>
@bors bors bot closed this as completed in 2399588 Dec 28, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

ExpDecay start step cossio/Flux.jl
3 participants
@ToucheSir @CarloLucibello @cossio