diff --git a/content/docs/command-reference/exp/init.md b/content/docs/command-reference/exp/init.md index f28bfeb665..6e78a2a353 100644 --- a/content/docs/command-reference/exp/init.md +++ b/content/docs/command-reference/exp/init.md @@ -128,11 +128,9 @@ $ dvc exp init './another_script.sh $MYENVVAR' your experiment can be found (if any). Overrides other configuration and default value (`plots/`). -- `--live` - configure the `path` directory for [DVCLive](/doc/dvclive). This is - where experiment logs will be written. Overrides other configuration and - default value (`dvclive/`). - - > This only has an effect when used with `--type=dl`. +- `--live` - set the path to the directory where the metrics and plots + [produced by DVCLive](https://dvc.org/doc/dvclive/dvclive-with-dvc#outputs) + will be found. - `--explicit` - do not assume default locations of project dependencies and outputs. You'll have to provide specific locations via other options or diff --git a/content/docs/command-reference/run.md b/content/docs/command-reference/run.md index 5f534eaee5..c5850bb3db 100644 --- a/content/docs/command-reference/run.md +++ b/content/docs/command-reference/run.md @@ -13,9 +13,7 @@ usage: dvc run [-h] [-q | -v] [-n ] [-f] [--outs-persist-no-cache ] [-m ] [-M ] [--plots ] [--plots-no-cache ] - [--live ] [--live-no-cache ] - [--live-no-html] [-w ] - [--always-changed] [--desc ] + [-w ] [--always-changed] [--desc ] [--no-exec] [--no-commit] [--no-run-cache] command @@ -269,16 +267,6 @@ $ dvc run -n second_stage './another_script.sh $MYENVVAR' - `--desc ` - user description of the stage (optional). This doesn't affect any DVC operations. -- `--live ` - specify the directory `path` for - [DVCLive](/doc/dvclive/dvclive-with-dvc) to write logs in. `path` will be - tracked (cached) by DVC. Saved in the `live` field of `dvc.yaml`. - -- `--live-no-cache ` - the same as `--live` except that the `path` is not - tracked by DVC. Useful if you prefer to track it with Git. - -- `--live-no-html` - deactivates DVCLive - [HTML report](/doc/dvclive/dvclive-with-dvc#html-report) generation. - - `-h`, `--help` - prints the usage/help message, and exit. - `-q`, `--quiet` - do not write anything to standard output. Exit with 0 if no diff --git a/content/docs/command-reference/stage/add.md b/content/docs/command-reference/stage/add.md index 218ead53d2..8fad58d475 100644 --- a/content/docs/command-reference/stage/add.md +++ b/content/docs/command-reference/stage/add.md @@ -12,9 +12,7 @@ usage: dvc stage add [-h] [-q | -v] -n [-f] [--outs-persist-no-cache ] [-m ] [-M ] [--plots ] [--plots-no-cache ] - [--live ] [--live-no-cache ] - [--live-no-html] [-w ] - [--always-changed] [--desc ] + [-w ] [--always-changed] [--desc ] command positional arguments: @@ -246,16 +244,6 @@ data science experiments. - `--desc ` - user description of the stage (optional). This doesn't affect any DVC operations. -- `--live ` - specify the directory `path` for - [DVCLive](/doc/dvclive/dvclive-with-dvc) to write logs in. `path` will be - tracked (cached) by DVC. Saved in the `live` field of `dvc.yaml`. - -- `--live-no-cache ` - the same as `--live` except that the `path` is not - tracked by DVC. Useful if you prefer to track it with Git. - -- `--live-no-html` - deactivates DVCLive - [HTML report](/doc/dvclive/dvclive-with-dvc#html-report) generation. - - `-h`, `--help` - prints the usage/help message, and exit. - `-q`, `--quiet` - do not write anything to standard output. Exit with 0 if no diff --git a/content/docs/dvclive/api-reference/live/index.md b/content/docs/dvclive/api-reference/live/index.md index 2a42c9a48c..8091103f47 100644 --- a/content/docs/dvclive/api-reference/live/index.md +++ b/content/docs/dvclive/api-reference/live/index.md @@ -9,6 +9,7 @@ class Live: self, path: Optional[str] = None, resume: bool = False, + report: Optional[str] = "html", ): ``` @@ -36,10 +37,12 @@ other metadata. - `dir` - Location of the directory to store [outputs](/doc/dvclive/get-started#outputs). -- `summary_path` - Location of the - [summary](/doc/dvclive/api-reference/live/#parameters). -- `html_path` - Location of the - [html report](/doc/dvclive/dvclive-with-dvc#html-report). + +- `summary_path` - `{dir}.json`. Location of the + [summary](/doc/dvclive/api-reference/live/log#description). + +- `html_path` - `{dir}/report.html`. Location of the + [html report](/doc/dvclive/api-reference/live/make_report#description). ## Parameters @@ -59,17 +62,18 @@ other metadata. -## Exceptions +- `report` - If `html`, DVCLive will call `Live.make_report()` on each step + update. _Default_: `html`. -- `dvclive.error.ConfigMismatchError` - thrown if the provided `path` does not - match with the one set in DVC (see - [DVCLive with DVC](/docs/dvclive/dvclive-with-dvc)) +- `auto_open` - If `True`, on the first `Live.make_report()` call, DVCLive will + automatically open `html_path` in a browser. _Default_: `False`. ## Methods - `Live.log()` - `Live.log_image()` - `Live.log_plot()` +- `Live.make_report()` - `Live.get_step()` - `Live.next_step()` - `Live.set_step()` diff --git a/content/docs/dvclive/api-reference/live/make_report.md b/content/docs/dvclive/api-reference/live/make_report.md new file mode 100644 index 0000000000..2f66156e7a --- /dev/null +++ b/content/docs/dvclive/api-reference/live/make_report.md @@ -0,0 +1,31 @@ +# Live.make_report() + +Generates an HTML Report from the logged data. + +```py +def make_report() +``` + +#### Usage: + +```py +from dvclive import Live + +live = Live() +live.log_plot("confusion_matrix", [0, 0, 1, 1], [1, 0, 0, 1]) +live.make_report() +``` + +## Description + +On each call, DVCLive will collect all the data logged in `{dir}`, generate an +HTML report and save it in `{dir}/report.html`. + +![](/img/dvclive-html.gif) + + + +This function gets called internally on each `step` update by default (unless +`report` is passed to `Live()` with a value other than `"html"`). + + diff --git a/content/docs/dvclive/api-reference/live/next_step.md b/content/docs/dvclive/api-reference/live/next_step.md index 7b4a30cbfe..2d05acafdf 100644 --- a/content/docs/dvclive/api-reference/live/next_step.md +++ b/content/docs/dvclive/api-reference/live/next_step.md @@ -28,17 +28,19 @@ You can use `Live.next_step()` to increase the `step` by 1 (one). Each metric logged in between `Live.next_step()` (or `Live.set_step()`) calls will be associated to the updated `step` value. + + +Each `Live.next_step()` will call `Live.make_report()` internally by default +(unless `report` is passed to `Live()` with a value other than `"html"`). + + + ### DVC integration When `dvclive` is used alongside `DVC`, each `Live.next_step()` call will have additional effects. -By default, on each `Live.next_step()` call, `DVC` will prepare an -[HTML report](/doc/dvclive/dvclive-with-dvc#html-report) with the -[metrics history](/doc/dvclive/get-started#history). - -In addition, when -[checkpoints](/doc/user-guide/experiment-management/checkpoints) are enabled in -the pipeline, `DVC` will +When [checkpoints](/doc/user-guide/experiment-management/checkpoints) are +enabled in the pipeline, DVC will [create a new checkpoint](/doc/dvclive/dvclive-with-dvc#checkpoints) on each `Live.next_step()` call. diff --git a/content/docs/dvclive/api-reference/live/set_step.md b/content/docs/dvclive/api-reference/live/set_step.md index be212d2cef..a5e3432fbb 100644 --- a/content/docs/dvclive/api-reference/live/set_step.md +++ b/content/docs/dvclive/api-reference/live/set_step.md @@ -29,18 +29,20 @@ You can use `Live.set_step()` to set `step` to any value. Each metric logged in between `Live.set_step()` (or `Live.next_step()`) calls will be associated to the provided `step` value. + + +Each `Live.set_step()` will call `Live.make_report()` internally by default +(unless `report` is passed to `Live()` with a value other than `"html"`). + + + ### DVC integration When `dvclive` is used alongside `DVC`, each `Live.set_step()` call will have additional effects. -By default, on each `Live.set_step()` call, `DVC` will prepare an -[HTML report](/doc/dvclive/dvclive-with-dvc#html-report) with the -[metrics history](/doc/dvclive/get-started#lhistory). - -In addition, when -[checkpoints](/doc/user-guide/experiment-management/checkpoints) are enabled in -the pipeline, `DVC` will +When [checkpoints](/doc/user-guide/experiment-management/checkpoints) are +enabled in the pipeline, DVC will [create a new checkpoint](/doc/dvclive/dvclive-with-dvc#checkpoints) on each `Live.set_step()` call. diff --git a/content/docs/dvclive/dvclive-with-dvc.md b/content/docs/dvclive/dvclive-with-dvc.md index 6837e16042..9759a1bb9a 100644 --- a/content/docs/dvclive/dvclive-with-dvc.md +++ b/content/docs/dvclive/dvclive-with-dvc.md @@ -1,15 +1,12 @@ # DVCLive with DVC -Even though DVCLive does not require DVC, they can integrate in several useful +Even though DVCLive does not require DVC, they can integrate in a couple useful ways: - The [outputs](#outputs) DVCLive produces are recognized by `dvc exp`, `dvc metrics` and `dvc plots`. Those same outputs can be visualized in [Iterative Studio](#iterative-studio). -- You can monitor model performance in realtime with the - [HTML report](#html-report) that DVCLive generates when used alongside DVC. - - DVCLive is also capable of generating [checkpoint](#checkpoints) signal files used by DVC experiments. @@ -17,12 +14,19 @@ ways: We will refer to a training script (`train.py`) already using `dvclive`: + + +If you use one of the supported [ML Frameworks](/doc/dvclive/ml-frameworks), you +can jump to its corresponding page to find an example usage. + + + ```python # train.py from dvclive import Live -live = Live() +live = Live("training_metrics") for epoch in range(NUM_EPOCHS): train_model(...) @@ -38,12 +42,23 @@ Let's use `dvc stage add` to create a stage to wrap this code (don't forget to `dvc init` first): ```dvc -$ dvc stage add -n train --live training_metrics - -d train.py python train.py +$ dvc stage add \ +--name train \ +--deps train.py \ +--metrics-no-cache training_metrics.json \ +--plots-no-cache training_metrics/scalars \ +python train.py ``` -`dvc.yaml` will contain a new `train` stage with the DVCLive configuration (in -the `live` field): + + +Note that the paths indicated in the `metrics` and `plots` options match the +`path` passed to `Live()` in the Python code (`"training_metrics"`). + + + +`dvc.yaml` will contain a new `train` stage using the DVCLive outputs as +`dvc metrics` and `dvc plots`: ```yaml stages: @@ -51,23 +66,14 @@ stages: cmd: python train.py deps: - train.py - live: - training_metrics: - summary: true - html: true + metrics: + - training_metrics.json: + cache: false + plots: + - training_metrics/scalars: + cache: false ``` -The value passed to `--live` (`training_metrics`) became the directory `path` -for DVCLive to write logs in, and DVC will now -[track](/doc/use-cases/versioning-data-and-model-files) it. Other supported -command options for the DVC integration: - -- `--live-no-cache ` - specify a DVCLive log directory `path` but don't - track it with DVC. Useful if you prefer to track it with Git. -- `--live-no-html` - deactivates [HTML report](#html-report) generation. - -> Note that DVC will not track summary files or the HTML report. - Run the training with `dvc repro` or `dvc exp run`: ```dvc @@ -83,6 +89,7 @@ $ tree ├── dvc.lock ├── dvc.yaml ├── training_metrics +│ ├── report.html │ └── scalars │ ├── acc.tsv │ └── loss.tsv @@ -90,13 +97,25 @@ $ tree └── train.py ``` -The [metrics summary](/doc/dvclive/api-reference/live/log#description) +The [metrics summary](/doc/dvclive/api-reference/live/log#description) in `training_metrics.json` can be used by `dvc metrics` and visualized with `dvc exp show`/`dvc exp diff`. -In addition, the -[metrics history](/doc/dvclive/api-reference/live/log#step-updates) generated -under `training_metrics/scalars` can be visualized with `dvc plots`. +The [metrics history](/doc/dvclive/api-reference/live/log#step-updates) +`training_metrics/scalars` can be visualized with `dvc plots`. + +The [HTML report](/doc/dvclive/api-reference/live/make_report#description) in +`training_metrics/report.html` will contain all the logged data and will be +automatically updated during training on each `step` update! + +![](/img/dvclive-html.gif) + + + +If you don't update the step number, the HTML report won't be generated unless +you call `Live.make_report()` directly. + + ### Iterative Studio @@ -107,21 +126,6 @@ experiments using DVCLive in Iterative Studio. ![](/img/dvclive-studio-plots.png) -### HTML report - -When `html: true`, DVC generates an _HTML report_. - -If you open `training_metrics_dvc_plots/index.html` in a browser, you'll see a -plot for the logged data automatically updated during the model training! - -![](/img/dvclive-html.gif) - - - -If you don't update the step number, the HTML report won't be generated. - - - ### Checkpoints When used alongside DVC, DVCLive can create _checkpoint_ signal files used by diff --git a/content/docs/dvclive/get-started.md b/content/docs/dvclive/get-started.md index 5952cc59c1..00f3ec38ed 100644 --- a/content/docs/dvclive/get-started.md +++ b/content/docs/dvclive/get-started.md @@ -109,6 +109,20 @@ not. See `Live.log()`, `Live.log_image()` and `Live.log_plot()` for more details. +### HTML report + +If and when `step` is updated, DVCLive generates or updates an HTML report in +`dvclive/report.html` which will contain all the logged data. + +![](/img/dvclive-html.gif) + + + +If you don't update the step number, the HTML report won't be generated unless +you call `Live.make_report()` directly. + + + ## What next? Learn how to use DVCLive alongside other tools: diff --git a/content/docs/sidebar.json b/content/docs/sidebar.json index 3b873c0ed1..13bc84d703 100644 --- a/content/docs/sidebar.json +++ b/content/docs/sidebar.json @@ -670,6 +670,10 @@ "slug": "log_plot", "label": "log_plot()" }, + { + "slug": "make_report", + "label": "make_report()" + }, { "slug": "get_step", "label": "get_step()" diff --git a/content/docs/user-guide/experiment-management/checkpoints.md b/content/docs/user-guide/experiment-management/checkpoints.md index 82b975b762..65ccdcf314 100644 --- a/content/docs/user-guide/experiment-management/checkpoints.md +++ b/content/docs/user-guide/experiment-management/checkpoints.md @@ -97,13 +97,11 @@ $ dvc stage add --name train \ --params seed,lr,weight_decay \ --checkpoints model.pt \ --plots-no-cache predictions.json \ - --live dvclive \ + --plots-no-cache dvclive/scalars \ + --metrics-no-cache dvclive.json python train.py ``` -💡 The `--live dvclive` option enables our special logger -[DVCLive](/doc/dvclive), which helps you register checkpoints from code. - The checkpoints need to be enabled in DVC at the pipeline level. The `-c / --checkpoint` option of the `dvc stage add` command defines the checkpoint file or directory. The checkpoint file, _model.pt_, is an output from one @@ -132,13 +130,14 @@ stages: outs: - model.pt: checkpoint: true + metrics: + - dvclive.json: + cache: false plots: + - dvclive/scalars: + cache: false - predictions.json: cache: false - live: - dvclive: - summary: true - html: true ``` ⚠️ Note that enabling checkpoints in a `dvc.yaml` file makes it incompatible diff --git a/content/docs/user-guide/project-structure/pipelines-files.md b/content/docs/user-guide/project-structure/pipelines-files.md index 674b6c906f..54fd433ed2 100644 --- a/content/docs/user-guide/project-structure/pipelines-files.md +++ b/content/docs/user-guide/project-structure/pipelines-files.md @@ -362,7 +362,6 @@ These are the fields that are accepted in each stage: | `always_changed` | Causes this stage to be always considered as [changed] by commands such as `dvc status` and `dvc repro`. `false` by default | | `meta` | (Optional) arbitrary metadata can be added manually with this field. Any YAML content is supported. `meta` contents are ignored by DVC, but they can be meaningful for user processes that read or write `.dvc` files directly. | | `desc` | (Optional) user description for this stage. This doesn't affect any DVC operations. | -| `live` | (Optional) [Dvclive](/doc/dvclive/dvclive-with-dvc) configuration field | [changed]: /doc/command-reference/status#local-workspace-status