Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add checkpoint backup #2672

Merged
merged 5 commits into from
Aug 20, 2021
Merged

Add checkpoint backup #2672

Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
d7920da
Add checkpoint backup
karajan1001 Jul 27, 2021
a09c508
Some updations
karajan1001 Aug 17, 2021
75321e9
rewording
casperdcl Aug 19, 2021
27f2274
Update content/docs/user-guide/experiment-management/checkpoints.md
jorgeorpinel Aug 20, 2021
7534c79
guide: fix formatting (Checkpoints)
jorgeorpinel Aug 20, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions content/docs/user-guide/experiment-management/checkpoints.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -473,3 +473,15 @@ Now that you know how to use checkpoints in DVC, you'll be able to resume
training from different checkpoints to try out new hyperparameters or code and
you'll be able to track all of the changes you make while trying to create the
best possible model.

## checkpoint backup

`DVC_EXP_AUTO_PUSH` env var is intended to help backup the experiments results
which helps to recover from interruptions that happened in the middle of a model
jorgeorpinel marked this conversation as resolved.
Show resolved Hide resolved
trainning. `DVC_EXP_AUTO_PUSH` can be set to a remote git repository address,
and then during the training process, remote backup will be done automatically
after every iteration. This includes Git references of experiments backup to
this repository and model caches, checkpoint files backup to the default dvc remote
jorgeorpinel marked this conversation as resolved.
Show resolved Hide resolved
cache.