Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DVCLive: Deprecate live section of dvc.yaml. #3411

Merged
merged 17 commits into from
May 12, 2022
Merged
8 changes: 3 additions & 5 deletions content/docs/command-reference/exp/init.md
Original file line number Diff line number Diff line change
Expand Up @@ -128,11 +128,9 @@ $ dvc exp init './another_script.sh $MYENVVAR'
your experiment can be found (if any). Overrides other configuration and
default value (`plots/`).

- `--live` - configure the `path` directory for [DVCLive](/doc/dvclive). This is
where experiment logs will be written. Overrides other configuration and
default value (`dvclive/`).

> This only has an effect when used with `--type=dl`.
- `--live` - set the path to the directory where the metrics and plots
[produced by DVCLive](https://dvc.org/doc/dvclive/dvclive-with-dvc#outputs)
will be found. Overrides other configuration and default value (`dvclive/`).

- `--explicit` - do not assume default locations of project dependencies and
outputs. You'll have to provide specific locations via other options or
Expand Down
14 changes: 1 addition & 13 deletions content/docs/command-reference/run.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,7 @@ usage: dvc run [-h] [-q | -v] [-n <name>] [-f]
[--outs-persist-no-cache <filename>]
[-m <path>] [-M <path>]
[--plots <path>] [--plots-no-cache <path>]
[--live <path>] [--live-no-cache <path>]
[--live-no-html] [-w <path>]
[--always-changed] [--desc <text>]
[-w <path>] [--always-changed] [--desc <text>]
[--no-exec] [--no-commit] [--no-run-cache]
command

Expand Down Expand Up @@ -269,16 +267,6 @@ $ dvc run -n second_stage './another_script.sh $MYENVVAR'
- `--desc <text>` - user description of the stage (optional). This doesn't
affect any DVC operations.

- `--live <path>` - specify the directory `path` for
[DVCLive](/doc/dvclive/dvclive-with-dvc) to write logs in. `path` will be
tracked (<abbr>cached</abbr>) by DVC. Saved in the `live` field of `dvc.yaml`.

- `--live-no-cache <path>` - the same as `--live` except that the `path` is not
tracked by DVC. Useful if you prefer to track it with Git.

- `--live-no-html` - deactivates DVCLive
[HTML report](/doc/dvclive/dvclive-with-dvc#html-report) generation.

- `-h`, `--help` - prints the usage/help message, and exit.

- `-q`, `--quiet` - do not write anything to standard output. Exit with 0 if no
Expand Down
14 changes: 1 addition & 13 deletions content/docs/command-reference/stage/add.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,7 @@ usage: dvc stage add [-h] [-q | -v] -n <name> [-f]
[--outs-persist-no-cache <filename>]
[-m <path>] [-M <path>]
[--plots <path>] [--plots-no-cache <path>]
[--live <path>] [--live-no-cache <path>]
[--live-no-html] [-w <path>]
[--always-changed] [--desc <text>]
[-w <path>] [--always-changed] [--desc <text>]
command

positional arguments:
Expand Down Expand Up @@ -246,16 +244,6 @@ data science experiments.
- `--desc <text>` - user description of the stage (optional). This doesn't
affect any DVC operations.

- `--live <path>` - specify the directory `path` for
[DVCLive](/doc/dvclive/dvclive-with-dvc) to write logs in. `path` will be
tracked (<abbr>cached</abbr>) by DVC. Saved in the `live` field of `dvc.yaml`.

- `--live-no-cache <path>` - the same as `--live` except that the `path` is not
tracked by DVC. Useful if you prefer to track it with Git.

- `--live-no-html` - deactivates DVCLive
[HTML report](/doc/dvclive/dvclive-with-dvc#html-report) generation.

- `-h`, `--help` - prints the usage/help message, and exit.

- `-q`, `--quiet` - do not write anything to standard output. Exit with 0 if no
Expand Down
21 changes: 13 additions & 8 deletions content/docs/dvclive/api-reference/live/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@ class Live:
self,
path: Optional[str] = None,
resume: bool = False,
report: Optional[str] = "html",
auto_open: bool = False,
daavoo marked this conversation as resolved.
Show resolved Hide resolved
):
```

Expand Down Expand Up @@ -36,10 +38,12 @@ other metadata.

- `dir` - Location of the directory to store
[outputs](/doc/dvclive/get-started#outputs).
- `summary_path` - Location of the
[summary](/doc/dvclive/api-reference/live/#parameters).
- `html_path` - Location of the
[html report](/doc/dvclive/dvclive-with-dvc#html-report).

- `summary_path` - `{dir}.json`. Location of the
[summary](/doc/dvclive/api-reference/live/log#description).

- `html_path` - `{dir}/report.html`. Location of the
[html report](/doc/dvclive/api-reference/live/make_report#description).

## Parameters

Expand All @@ -59,17 +63,18 @@ other metadata.

</admon>

## Exceptions
- `report` - If `html`, DVCLive will call `Live.make_report()` on each step
update. _Default_: `html`.

- `dvclive.error.ConfigMismatchError` - thrown if the provided `path` does not
match with the one set in DVC (see
[DVCLive with DVC](/docs/dvclive/dvclive-with-dvc))
- `auto_open` - If `True`, on the first `Live.make_report()` call, DVCLive will
automatically open `html_path` in a browser. _Default_: `False`.

## Methods

- `Live.log()`
- `Live.log_image()`
- `Live.log_plot()`
- `Live.make_report()`
- `Live.get_step()`
- `Live.next_step()`
- `Live.set_step()`
31 changes: 31 additions & 0 deletions content/docs/dvclive/api-reference/live/make_report.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# Live.make_report()

Generates an HTML Report from the logged data.

```py
def make_report()
```

#### Usage:
jorgeorpinel marked this conversation as resolved.
Show resolved Hide resolved

```py
from dvclive import Live

live = Live()
live.log_plot("confusion_matrix", [0, 0, 1, 1], [1, 0, 0, 1])
live.make_report()
```

## Description

On each call, DVCLive will collect all the data logged in `{dir}`, generate an
HTML report and save it in `{dir}/report.html`.

![](/img/dvclive-html.gif)

<admon type="info">

This function gets automatically called on each `step` update, if
`report="html"` was passed to `Live()`
daavoo marked this conversation as resolved.
Show resolved Hide resolved

</admon>
16 changes: 9 additions & 7 deletions content/docs/dvclive/api-reference/live/next_step.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,17 +28,19 @@ You can use `Live.next_step()` to increase the `step` by 1 (one).
Each metric logged in between `Live.next_step()` (or `Live.set_step()`) calls
will be associated to the updated `step` value.

<admon type="info">

Each `Live.next_step()` will call `Live.make_report()` internally by default
(unless `report` is passed to `Live()` with a value other than `"html"`).

</admon>

### DVC integration

When `dvclive` is used alongside `DVC`, each `Live.next_step()` call will have
additional effects.

By default, on each `Live.next_step()` call, `DVC` will prepare an
[HTML report](/doc/dvclive/dvclive-with-dvc#html-report) with the
[metrics history](/doc/dvclive/get-started#history).

In addition, when
[checkpoints](/doc/user-guide/experiment-management/checkpoints) are enabled in
the <abbr>pipeline</abbr>, `DVC` will
When [checkpoints](/doc/user-guide/experiment-management/checkpoints) are
enabled in the <abbr>pipeline</abbr>, DVC will
[create a new checkpoint](/doc/dvclive/dvclive-with-dvc#checkpoints) on each
`Live.next_step()` call.
16 changes: 9 additions & 7 deletions content/docs/dvclive/api-reference/live/set_step.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,18 +29,20 @@ You can use `Live.set_step()` to set `step` to any value.
Each metric logged in between `Live.set_step()` (or `Live.next_step()`) calls
will be associated to the provided `step` value.

<admon type="info">

Each `Live.set_step()` will call `Live.make_report()` internally by default
(unless `report` is passed to `Live()` with a value other than `"html"`).

</admon>

### DVC integration

When `dvclive` is used alongside `DVC`, each `Live.set_step()` call will have
additional effects.

By default, on each `Live.set_step()` call, `DVC` will prepare an
[HTML report](/doc/dvclive/dvclive-with-dvc#html-report) with the
[metrics history](/doc/dvclive/get-started#lhistory).

In addition, when
[checkpoints](/doc/user-guide/experiment-management/checkpoints) are enabled in
the <abbr>pipeline</abbr>, `DVC` will
When [checkpoints](/doc/user-guide/experiment-management/checkpoints) are
enabled in the <abbr>pipeline</abbr>, DVC will
[create a new checkpoint](/doc/dvclive/dvclive-with-dvc#checkpoints) on each
`Live.set_step()` call.

Expand Down
90 changes: 47 additions & 43 deletions content/docs/dvclive/dvclive-with-dvc.md
Original file line number Diff line number Diff line change
@@ -1,28 +1,32 @@
# DVCLive with DVC

Even though DVCLive does not require DVC, they can integrate in several useful
Even though DVCLive does not require DVC, they can integrate in a couple useful
ways:

- The [outputs](#outputs) DVCLive produces are recognized by `dvc exp`,
`dvc metrics` and `dvc plots`. Those same outputs can be visualized in
[Iterative Studio](#iterative-studio).

- You can monitor model performance in realtime with the
[HTML report](#html-report) that DVCLive generates when used alongside DVC.

- DVCLive is also capable of generating [checkpoint](#checkpoints) signal files
jorgeorpinel marked this conversation as resolved.
Show resolved Hide resolved
used by DVC <abbr>experiments<abbr>.

## Setup

We will refer to a training script (`train.py`) already using `dvclive`:

<admon type="tip">

If you use one of the supported [ML Frameworks](/doc/dvclive/ml-frameworks), you
can jump to its corresponding page to find an example usage.
jorgeorpinel marked this conversation as resolved.
Show resolved Hide resolved

</admon>

```python
# train.py

from dvclive import Live

live = Live()
live = Live("training_metrics")

for epoch in range(NUM_EPOCHS):
train_model(...)
Expand All @@ -38,36 +42,38 @@ Let's use `dvc stage add` to create a stage to wrap this code (don't forget to
`dvc init` first):

```dvc
$ dvc stage add -n train --live training_metrics
-d train.py python train.py
$ dvc stage add \
--name train \
--deps train.py \
--metrics-no-cache training_metrics.json \
--plots-no-cache training_metrics/scalars \
python train.py
```

`dvc.yaml` will contain a new `train` stage with the DVCLive configuration (in
the `live` field):
<admon type="exclamation">
daavoo marked this conversation as resolved.
Show resolved Hide resolved

Note that the paths indicated in the `metrics` and `plots` options match the
`path` passed to `Live()` in the Python code (`"training_metrics"`).

</admon>

`dvc.yaml` will contain a new `train` stage using the DVCLive outputs as
`dvc metrics` and `dvc plots`:

```yaml
stages:
train:
cmd: python train.py
deps:
- train.py
live:
training_metrics:
summary: true
html: true
metrics:
- training_metrics.json:
cache: false
plots:
- training_metrics/scalars:
cache: false
```

The value passed to `--live` (`training_metrics`) became the directory `path`
for DVCLive to write logs in, and DVC will now
[track](/doc/use-cases/versioning-data-and-model-files) it. Other supported
command options for the DVC integration:

- `--live-no-cache <path>` - specify a DVCLive log directory `path` but don't
track it with DVC. Useful if you prefer to track it with Git.
- `--live-no-html` - deactivates [HTML report](#html-report) generation.

> Note that DVC will not track summary files or the HTML report.

Run the training with `dvc repro` or `dvc exp run`:

```dvc
Expand All @@ -83,20 +89,33 @@ $ tree
├── dvc.lock
├── dvc.yaml
├── training_metrics
│ ├── report.html
│ └── scalars
│ ├── acc.tsv
│ └── loss.tsv
├── training_metrics.json
└── train.py
```

The [metrics summary](/doc/dvclive/api-reference/live/log#description)
The [metrics summary](/doc/dvclive/api-reference/live/log#description) in
`training_metrics.json` can be used by `dvc metrics` and visualized with
daavoo marked this conversation as resolved.
Show resolved Hide resolved
`dvc exp show`/`dvc exp diff`.

In addition, the
[metrics history](/doc/dvclive/api-reference/live/log#step-updates) generated
under `training_metrics/scalars` can be visualized with `dvc plots`.
The [metrics history](/doc/dvclive/api-reference/live/log#step-updates)
`training_metrics/scalars` can be visualized with `dvc plots`.

The [HTML report](/doc/dvclive/api-reference/live/make_report#description) in
`training_metrics/report.html` will contain all the logged data and will be
automatically updated during training on each `step` update!

![](/img/dvclive-html.gif)

<admon type="info">

If you don't update the step number, the HTML report won't be generated unless
you call `Live.make_report()` directly.

</admon>

### Iterative Studio

Expand All @@ -107,21 +126,6 @@ experiments using DVCLive in Iterative Studio.

![](/img/dvclive-studio-plots.png)

### HTML report

When `html: true`, DVC generates an _HTML report_.

If you open `training_metrics_dvc_plots/index.html` in a browser, you'll see a
plot for the logged data automatically updated during the model training!

![](/img/dvclive-html.gif)

<admon type="info">

If you don't update the step number, the HTML report won't be generated.

</admon>

### Checkpoints

When used alongside DVC, DVCLive can create _checkpoint_ signal files used by
Expand Down
14 changes: 14 additions & 0 deletions content/docs/dvclive/get-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,20 @@ not.

See `Live.log()`, `Live.log_image()` and `Live.log_plot()` for more details.

### HTML report

If and when `step` is updated, DVCLive generates or updates an HTML report in
`dvclive/report.html` which will contain all the logged data.
jorgeorpinel marked this conversation as resolved.
Show resolved Hide resolved

![](/img/dvclive-html.gif)

<admon type="info">

If you don't update the step number, the HTML report won't be generated unless
you call `Live.make_report()` directly.

</admon>

## What next?

Learn how to use DVCLive alongside other tools:
Expand Down
4 changes: 4 additions & 0 deletions content/docs/sidebar.json
Original file line number Diff line number Diff line change
Expand Up @@ -670,6 +670,10 @@
"slug": "log_plot",
"label": "log_plot()"
},
{
"slug": "make_report",
"label": "make_report()"
},
{
"slug": "get_step",
"label": "get_step()"
Expand Down
Loading