Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add checkpoint backup #2672

Merged
merged 5 commits into from
Aug 20, 2021
Merged

Conversation

karajan1001
Copy link
Contributor

@karajan1001 karajan1001 commented Jul 27, 2021

@karajan1001 karajan1001 marked this pull request as draft July 27, 2021 12:43
@jorgeorpinel jorgeorpinel added the ⌛ status: wait-core-merge Waiting for related product PR merge/release label Jul 28, 2021
@iterative iterative deleted a comment from karajan1001 Jul 28, 2021
@jorgeorpinel

This comment has been minimized.

@karajan1001

This comment has been minimized.

@jorgeorpinel

This comment has been minimized.

@karajan1001

This comment has been minimized.

@karajan1001
Copy link
Contributor Author

karajan1001 commented Aug 13, 2021

The core PR had been merged, we can continue on this one. For now we have two envs:

  1. DVC_EXP_GIT_REMOTE: Remote Git repo for the exp refs to back up. Can be a URL or a name ( origin, myremote, etc). This can be used for default git remote in dvc exp push
  2. DVC_EXP_AUTO_PUSH: trigger of the auto push. (true, 1, y, yes) for Ture.

Other things to be noticed:

  1. There is a validation in the beginning of the iteration, if one of DVC_EXP_GIT_REMOTE and dvc default remote cache is not setted , it will raise exception and error out.
  2. But if the validation passed and the auto push fails during the iterations it will only give a warning and continue the process.
  3. The DVC_EXP_GIT_REMOTE can be set to the running repo. In this case cache will be pushed to dvc remote and exp refs will not (it is already in the running repo).

What else is needed? Some examples or other things?

@casperdcl casperdcl added A: docs Area: user documentation (gatsby-theme-iterative) and removed ⌛ status: wait-core-merge Waiting for related product PR merge/release labels Aug 13, 2021
@karajan1001 karajan1001 marked this pull request as ready for review August 17, 2021 03:25
@karajan1001
Copy link
Contributor Author

Not good at English expression need some help.

@iesahin
Copy link
Contributor

iesahin commented Aug 17, 2021

谢谢 @karajan1001 Taking this over.

@iesahin
Copy link
Contributor

iesahin commented Aug 17, 2021

Ah, excuse me, @casperdcl already assigned this to himself. Thank you :)

@casperdcl casperdcl self-requested a review August 17, 2021 20:09
Copy link
Contributor

@casperdcl casperdcl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reworded; keeping header for #id backlink user convenience/referencing.

be pushed. With this configuration, `dvc exp push` will be done automatically
after every iteration.

⚠️ If either Git or DVC remotes are missing, the experiment will fail. However,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can split this into two warnings.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are the 2 warnings?

@jorgeorpinel
Copy link
Contributor

Where is Resyled when you need it 😢 it's been acting up recently...

@jorgeorpinel jorgeorpinel merged commit 2888683 into iterative:master Aug 20, 2021
@karajan1001 karajan1001 deleted the checkpoint_auto_push branch August 22, 2021 13:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A: docs Area: user documentation (gatsby-theme-iterative)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants