checkpoints: flag to exp push && push

[casperdcl]

Provide a flag (e.g. dvclive.next_step(push=True) or make_checkpoint(push=True) or DVC_EXP_AUTO_PUSH=<git_remote_name_or_url|true>) to push things per-checkpoint.

Helps with resuming a long-running CI job after timeout.

Part of iterative/cml#560.

[karajan1001]

@efiop, Some analysis after looking into it.

1. This is a DVC only option. Shouldn't do any change in dvclive which would make it rely on DVC.
2. It would be running after every turn of the stage running,  we can create a new monitor task for it like other dvclive ones for it.
3. A new parameter in the stage file is needed to control this process.

Any suggestions for this?

[dberenbaum]

Can this be an argument to make_checkpoint()?

[pmrowla]

Given that this is a CI thing, it seems like maybe it should just be set through an env var?

something like like DVC_EXP_AUTO_PUSH=<git_remote_name_or_url>

[pmrowla]

Also note that for CML purposes, I think we will want both DVC cache and run-cache to be included with the automated exp push (DVC cache is already included by default on exp push but run cache is normally optional)

[dberenbaum]

Is this going to quickly blow up the number of exp refs on Github? Do we need some mechanism to delete once merged/closed?

Related:
#6006
#6329 (distant cousin 😄 )

[casperdcl]

dvc exp gc --checkpoints?

[dberenbaum]

I think there's a few potential issues:

1. For experiments with many checkpoints, I may want to squash the checkpoint commits so that we only keep the final checkpoint results, which I'm assuming is what you meant with dvc exp gc --checkpoints @casperdcl?
2. For any experiment cml-pr that I close because the performance is bad, there's a bug, or any other reason, I may want to delete the experiment reference entirely.
3. Similarly, for any experiment cml-pr that I merge, I may want to delete the experiment reference since it's now duplicated (this is the connection I was thinking wrt exp branch: Add a flag --remove or --delete to remove the persisted experiment  #6329).

The latter two could be automatically handled in the cml workflow, or they could be handled with dvc exp gc --cloud or dvc exp gc origin to occasionally clean up as needed. These would also drop the experiment references for the checkpoints mentioned in the first issue, but the commits would still be in the merged branch unless we have some way to squash them before merging.

[pmrowla]

Is this going to quickly blow up the number of exp refs on Github?

In terms of "will this break git/github", this is not an issue, there's no real limit to the number of refs git can handle, and the server will automatically pack them as needed.

But yes, it will inflate the number of exps that would show up when users use exp list w/their server URL

Do we need some mechanism to delete once merged/closed?

These commits go into the CML PR once the run is completed, correct? At that point, it seems to me that the exp ref is no longer needed and can just be removed (it doesn't matter whether or not the PR has been merged yet, the PR is it's own git ref).

These would also drop the experiment references for the checkpoints mentioned in the first issue, but the commits would still be in the merged branch unless we have some way to squash them before merging.

If PRs are being merged on the github side, github already provides the option to squash before merging

[dberenbaum]

Great, so it seems like all that's needed is a mechanism to delete the exp refs once the pr is ready?