Checkpoints example

[dberenbaum]

There is no ML example of checkpoints in the docs, so I started a simple example repo as a potential project to link to in the docs: https://github.com/iterative/dvc-checkpoints-mnist.

There are two branches:

• minimal has about as simple a DVC project as I could make to showcase checkpoints with as few other DVC features complicating it.
• main is a more fleshed out DVC project that includes multiple stages, params, plots, etc.

Interested in feedback on:

1. Would it be useful to link to something like this in the docs (even if the specifics of the repo could use some work)?
2. Which branch seems more useful?
3. General feedback on the readme, project structure, etc.

@dmpetrov @shcheklein @jorgeorpinel @pmrowla

[shcheklein]

My thoughts on this - yes, it is very useful. It would be great to do a high level Use Case about experiments and do a tutorial about them, similar to the basic data versioning one.

[dberenbaum]

Thanks, that sounds like a good goal. Keep in mind that this is specifically for checkpoints, not experiments generally. I think it will be rare to use checkpoints and experiment with other parameters simultaneously, although some may do some experimentation and then use checkpoints to optimize the final model.

[dberenbaum]

A couple of additional thoughts:

1. I added a 'live' branch with dvclive integration.
2. Do we want to include a link for the release and then iterate later into something like a tutorial?

[jorgeorpinel]

I added a 'live' branch with dvclive integration.

the main branch seems to combine Dvclive and DVC checkpoints which is somewhat confusing (in the README at least) and I think ill-advised (per some conversation we had with @pared I think)? But I need to try each branch to get a better understanding ⌛

In general I think that definitely an example repo for experiments i super helpful but not sure that it needs to be specifically about checkpoints.

[pared]

@jorgeorpinel I think that was the initial point of this repo. To provide example for checkpoints/live interoperability.
Though if we would like to maintain some order of learinging, mybe it makes sense to make minimal the default branch and let user checkout the live?

[dberenbaum]

Initially, the repo was intended to show basic checkpoints functionality. After some more thought and looking through the docs, it made more sense to have it illustrate dvclive. The rationale behind that decision was:

1. dvclive and checkpoints are generally useful in similar scenarios. Both are tracking model progress over time.
2. dvclive has added benefits when integrating with a dvc checkpoints project, like automatically writing out the metrics and providing a real-time logger. We might even want to consider encouraging anyone using checkpoints to use dvclive or automatically integrating it into checkpoints stages in the future.
3. The docs for checkpoints already have a decent example of how to add checkpoints to a Python stage, but there is no example of how to integrate dvclive into a stage like this.