Stopping criteria #227
Labels
Comments
Closed
bors bot
added a commit
that referenced
this issue
Apr 28, 2021
1545: Add early stopping utils r=darsnack a=queensferryme This pull request introduces a utility function `early_stopping(f; min_delta=0, patience=3)` for conveniently implementing early stopping. Its motive was discussed in #227 back in 2018. Also AFAIU, early stopping is widely adopted in training large-scale neural networks nowadays as a mechanism to prevent overfitting. Therefore, I believe a built-in utility function for early stopping would be beneficial for Flux.jl users. The implementation is heavily based on [tf.keras.callbacks.EarlyStopping](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/EarlyStopping) and [ignite.handlers.EarlyStopping](https://github.com/pytorch/ignite/blob/master/ignite/handlers/early_stopping.py). > This is a _draft_ implementation. Documentation and tests are not added yet. As I am new to Flux.jl, I think advice and opinions from maintainers would be helpful before I proceed. **Example Usage**: ```julia # test_ea.jl using Flux function loss() v = 1 return () -> v += 1 end ea = Flux.early_stopping(loss(), patience=5) Flux.@epochs 10 begin ea() || break end ``` **Output**: ``` ➜ julia test_ea.jl Activating environment at `~/Developer/Flux.jl/Project.toml` [ Info: Epoch 1 [ Info: Epoch 2 [ Info: Epoch 3 [ Info: Epoch 4 [ Info: Epoch 5 [ Info: Epoch 6 ➜ ``` ### PR Checklist - [x] Tests are added - [ ] Entry in NEWS.md - [x] Documentation, if applicable - [ ] API changes require approval from a committer (different from the author, if applicable) Co-authored-by: Queensferry <[email protected]> Co-authored-by: Dhairya Gandhi <[email protected]>
bors bot
added a commit
that referenced
this issue
May 14, 2021
1545: Add early stopping utils r=darsnack a=queensferryme This pull request introduces a utility function `early_stopping(f; min_delta=0, patience=3)` for conveniently implementing early stopping. Its motive was discussed in #227 back in 2018. Also AFAIU, early stopping is widely adopted in training large-scale neural networks nowadays as a mechanism to prevent overfitting. Therefore, I believe a built-in utility function for early stopping would be beneficial for Flux.jl users. The implementation is heavily based on [tf.keras.callbacks.EarlyStopping](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/EarlyStopping) and [ignite.handlers.EarlyStopping](https://github.com/pytorch/ignite/blob/master/ignite/handlers/early_stopping.py). > This is a _draft_ implementation. Documentation and tests are not added yet. As I am new to Flux.jl, I think advice and opinions from maintainers would be helpful before I proceed. **Example Usage**: ```julia # test_ea.jl using Flux function loss() v = 1 return () -> v += 1 end ea = Flux.early_stopping(loss(), patience=5) Flux.@epochs 10 begin ea() || break end ``` **Output**: ``` ➜ julia test_ea.jl Activating environment at `~/Developer/Flux.jl/Project.toml` [ Info: Epoch 1 [ Info: Epoch 2 [ Info: Epoch 3 [ Info: Epoch 4 [ Info: Epoch 5 [ Info: Epoch 6 ➜ ``` ### PR Checklist - [x] Tests are added - [ ] Entry in NEWS.md - [x] Documentation, if applicable - [ ] API changes require approval from a committer (different from the author, if applicable) Co-authored-by: Queensferry <[email protected]> Co-authored-by: Dhairya Gandhi <[email protected]>
|
Looks like #1545 has captured most of this. The rest is arguably better suited for an external package like https://github.com/JuliaAI/EarlyStopping.jl. |
|
Yeah I think this is closed by #1545. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
It came up in #131 and #132,
I think we should have a number of predefined stopping callbacks,
So that you feed the training data in as an an infintity stream of epochs.
And you have stop based on criteria.
The ones I am thinking are:
atolstop when the loss is below some point (obv. 0 as default)rtolstop when the decrease in loss from one epoch to the next is below some point (obv. 0 as default)atolandrtolcould be parameterised to use different loss functions,including different datasets. Early stopping can be achieved by using a development dataset instead of the training dataset.
We could have more; e.g. incorporating some from of hysteresis to for looking at improvement over n-epochs, like some kind of
rⁿtolbut that can be later.Of course the user should also still be able to define their own custom methods.
The text was updated successfully, but these errors were encountered: