diff --git a/content/docs/sidebar.json b/content/docs/sidebar.json index 676f943a01..f63d316db3 100644 --- a/content/docs/sidebar.json +++ b/content/docs/sidebar.json @@ -61,6 +61,7 @@ }, { "slug": "experiments", + "label": "Experiments", "tutorials": { "katacoda": "https://katacoda.com/dvc/courses/get-started/experiments" } diff --git a/content/docs/start/experiments.md b/content/docs/start/experiments.md index 13568927da..c23058b319 100644 --- a/content/docs/start/experiments.md +++ b/content/docs/start/experiments.md @@ -2,293 +2,237 @@ title: 'Get Started: Experiments' --- -# Get Started: Experiments +# Get Started with Experiments -_New in DVC 2.0_ - -<abbr>Experiments</abbr> proliferate quickly in ML projects where there are many -parameters to tune or other permutations of the code. We can organize such -projects and keep only what we ultimately need with `dvc exp` commands. DVC can -track experiments for you so there's no need to commit each one to Git. This way -your repo doesn't become polluted with all of them. You can discard experiments -once they're no longer needed. - -> ๐ See [Experiment Management](/doc/user-guide/experiment-management) for more -> information on DVC's approach. +In machine learning projects, the number of <abbr>experiments</abbr> grows +rapidly. DVC can track these experiments, list and compare their most relevant +parameters and metrics, navigate among them, and commit only the ones that we +need to Git. https://youtu.be/FHQq_zZz5ms -## Running experiments +In this section, we explore the basic features of DVC experiment management with +the [`example-dvc-experiments`][ede] project. -Previously, we learned how to tune [ML pipelines](/doc/start/data-pipelines) and -[compare the changes](/doc/start/metrics-parameters-plots). Let's further -increase the number of features in the `featurize` stage to see how it compares. +[ede]: https://github.com/iterative/example-dvc-experiments -`dvc exp run` makes it easy to change <abbr>hyperparameters</abbr> and run a new -experiment: +<details> -```dvc -$ dvc exp run --set-param featurize.max_features=3000 -``` +### โ๏ธ Installing the example project -<details> +These commands are run in the [`example-dvc-experiments`][ede] project. You can +run the commands in this document after cloning the repository, installing the +requirements, and pulling the data. -### ๐ก Expand to see what happens under the hood. +#### Clone the project and create virtual environment -`dvc exp run` is similar to `dvc repro` but with some added conveniences for -running experiments. The `--set-param` (or `-S`) flag sets the values for -<abbr>parameters<abbr> as a shortcut for editing `params.yaml`. +Please clone the project and create a virtual environment. -Check that the `featurize.max_features` value has been updated in `params.yaml`: +> We strongly recommend to create a virtual environment to keep the libraries we +> use isolated from the rest of your system. This prevents version conflicts. -```git - featurize: -- max_features: 1500 -+ max_features: 3000 +```dvc +$ git clone https://github.com/iterative/example-dvc-experiments -b get-started +$ cd example-dvc-experiments +$ virtualenv .venv +$ . .venv/bin/activate +$ python -m pip install -r requirements.txt ``` -Any edits to <abbr>dependencies</abbr> (parameters or source code) will be -reflected in the experiment run. +#### Get the data set -</details> - -`dvc exp diff` compares experiments: +The repository we cloned doesn't contain the dataset. Instead of storing the +data in the Git repository, we use DVC to retrieve from a shared data store. In +this case, we use `dvc pull` to update the missing data files. ```dvc -$ dvc exp diff -Path Metric Value Change -scores.json avg_prec 0.56191 0.009322 -scores.json roc_auc 0.93345 0.018087 - -Path Param Value Change -params.yaml featurize.max_features 3000 1500 +$ dvc pull ``` -## Queueing experiments +The repository already contains the necessary configuration to run the +experiments. -So far, we have been tuning the `featurize` stage, but there are also parameters -for the `train` stage (which trains a -[random forest classifier](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html)). - -These are the `train` parameters from `params.yaml`: - -```yaml -train: - seed: 20170428 - n_est: 50 - min_split: 2 -``` +</details> -Let's set up experiments with different hyperparameters. We can use the -`--queue` flag to define all the combinations we want to try without executing -anything (yet): +Running the experiment with the default project settings requires only the +command: ```dvc -$ dvc exp run --queue -S train.min_split=8 -Queued experiment 'd3f6d1e' for future execution. -$ dvc exp run --queue -S train.min_split=64 -Queued experiment 'f1810e0' for future execution. -$ dvc exp run --queue -S train.min_split=2 -S train.n_est=100 -Queued experiment '7323ea2' for future execution. -$ dvc exp run --queue -S train.min_split=8 -S train.n_est=100 -Queued experiment 'c605382' for future execution. -$ dvc exp run --queue -S train.min_split=64 -S train.n_est=100 -Queued experiment '0cdee86' for future execution. +$ dvc exp run +... +Reproduced experiment(s): exp-b28f0 +Experiment results have been applied to your workspace. +... ``` -Next, run all (`--run-all`) queued experiments in parallel (using `--jobs`): +It runs the specified command (`python train.py`) in `dvc.yaml`. That command +writes the metrics values to `metrics.json`. -```dvc -$ dvc exp run --run-all --jobs 2 -``` +This experiment is then associated with the values found in the parameters file +(`params.yaml`), and other dependencies (`data/images/`) with these produced +metrics. -## Comparing many experiments +The purpose of the `dvc exp` family of commands is to let you run, capture, and +compare the machine learning experiments at once as you iterate on your project. +The artifacts like models and metrics produced by each experiment are tracked by +DVC, and the associated parameters and metrics can be committed to Git as text +files. -To compare all of these experiments, we need more than `diff`. `dvc exp show` -compares any number of experiments in one table: +You can review the experiment results with `dvc exp show` and see these metrics +and results in a nicely formatted table: ```dvc -$ dvc exp show --no-timestamp \ - --include-params train.n_est,train.min_split +$ dvc exp show ``` ```dvctable -โโโโโโโโโโโโโโโโโณโโโโโโโโโโโณโโโโโโโโโโณโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโ -โ neutral:**Experiment** โ metric:**avg_prec** โ metric:**roc_auc** โ param:**train.n_est**โ param:**train.min_split** โ -โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ -โ **workspace** โ **0.56191** โ **0.93345** โ **50** โ **2** โ -โ **master** โ **0.55259** โ **0.91536** โ **50** โ **2** โ -โ โโโ exp-bfe64 โ 0.57833 โ 0.95555 โ 50 โ 8 โ -โ โโโ exp-b8082 โ 0.59806 โ 0.95287 โ 50 โ 64 โ -โ โโโ exp-c7250 โ 0.58876 โ 0.94524 โ 100 โ 2 โ -โ โโโ exp-b9cd4 โ 0.57953 โ 0.95732 โ 100 โ 8 โ -โ โโโ exp-98a96 โ 0.60405 โ 0.9608 โ 100 โ 64 โ -โ โโโ exp-ad5b1 โ 0.56191 โ 0.93345 โ 50 โ 2 โ -โโโโโโโโโโโโโโโโโดโโโโโโโโโโโดโโโโโโโโโโดโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโ +โโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโณโโโโโโโโโโณโโโโโโโโโณโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโ +โ white:**Experiment** โ white:**Created** โ yellow:**loss** โ yellow:**acc** โ blue:**train.epochs** โ blue:**model.conv_units** โ +โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ +โ workspace โ - โ 0.23282 โ 0.9152 โ 10 โ 16 โ +โ 7317bc6 โ Jul 18, 2021 โ - โ - โ 10 โ 16 โ +โ โโโ 1a1d858 [exp-6dccf] โ 03:21 PM โ 0.23282 โ 0.9152 โ 10 โ 16 โ +โโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโดโโโโโโโโโโดโโโโโโโโโดโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโ ``` -Each experiment is given an arbitrary name by default (although we can specify -one with `dvc exp run -n`.) We can see that `exp-98a96` performed best among -both of our metrics, with 100 estimators and a minimum of 64 samples to split a -node. - -> See `dvc exp show --help` for more info on its options. - -## Persisting experiments - -Now that we know the best parameters, let's keep that experiment and ignore the -rest. - -`dvc exp apply` rolls back the <abbr>workspace<abbr> to the specified -experiment: - -```dvc -$ dvc exp apply exp-98a96 -Changes for experiment 'exp-98a96' have been applied to your workspace. -``` +The `workspace` row in the table shows the results of the most recent experiment +that's available in the <abbr>workspace</abbr>. The table also shows each +experiment in a separate row, along with the Git commit IDs they are attached +to. We can see that the experiment we run has a name `exp-6dccf` and was run +from the commit ID `7317bc6`. <details> -### ๐ก Expand to see what happens under the hood. +### โน๏ธ If you used `dvc repro` before -`dvc exp apply` is similar to `dvc checkout`, but works with experiments -instead. DVC tracks everything in the pipeline for each experiment (parameters, -metrics, dependencies, and outputs), retrieving things later as needed. +Earlier versions of DVC uses `dvc repro` to run the pipeline. If you already +have a DVC project, you may already be using `dvc repro`. -Check that `scores.json` reflects the metrics in the table above: +We use `dvc repro` to run the pipeline as found in the <abbr>workspace</abbr>. +All the parameters and dependencies are retrieved from the current workspace. It +doesn't use any specialized mechanism to track experiments. -```json -{ "avg_prec": 0.6040544652105823, "roc_auc": 0.9608017142900953 } -``` +When you have a large number of experiments that you don't want to commit all to +Git, it's better to use `dvc exp run`. It allows to change the parameters +quickly, can track the history of artifacts and has facilities to compare these +experiments easily. </details> -Once an experiment has been applied to the workspace, it is no different from -reproducing the result without `dvc exp run`. Let's make it persistent in our -regular pipeline by committing it in our Git branch: - -```dvc -$ git add dvc.lock params.yaml prc.json roc.json scores.json -$ git commit -a -m "Preserve best random forest experiment" -``` - -## Sharing experiments +Now let's do some more experimentation. -After committing the best experiments to our Git branch, we can -[store and share](/doc/start/data-and-model-versioning#storing-and-sharing) them -remotely like any other iteration of the pipeline. +DVC allows to update the parameters defined in the pipeline without modifying +the files manually. We use this feature to set the convolutional units in +`train.py`. ```dvc -dvc push -git push +$ dvc exp run --set-param model.conv_units=24 +... +Reproduced experiment(s): exp-7b56f +Experiment results have been applied to your workspace. +... ``` <details> -### ๐ก Important information on storing experiments remotely. - -The commands in this section require both a `dvc remote default` and a -[Git remote](https://git-scm.com/book/en/v2/Git-Basics-Working-with-Remotes). A -DVC remote stores the experiment data, and a Git remote stores the code, -parameters, and other metadata associated with the experiment. DVC supports -various types of remote storage (local file system, SSH, Amazon S3, Google Cloud -Storage, HTTP, HDFS, etc.). The Git remote is often a central Git server -(GitHub, GitLab, BitBucket, etc.). +### โ๏ธ Run multiple experiments in parallel -</details> - -Experiments that have not been made persistent will not be stored or shared -remotely through `dvc push` or `git push`. +Instead of running the experiments one-by-one, we can define them to run in a +batch. This is especially handy when you have long running experiments. -`dvc exp push` enables storing and sharing any experiment remotely. +We add experiments to the queue using the `--queue` option of `dvc exp run`. We +also use `-S` (`--set-param`) to set a value for the parameter. ```dvc -$ dvc exp push gitremote exp-bfe64 -Pushed experiment 'exp-bfe64' to Git remote 'gitremote'. +$ dvc exp run --queue -S model.conv_units=32 +Queued experiment '3cac8c6' for future execution. +$ dvc exp run --queue -S model.conv_units=64 +Queued experiment '23660b6' for future execution. +$ dvc exp run --queue -S model.conv_units=128 +Queued experiment '6591a57' for future execution. +$ dvc exp run --queue -S model.conv_units=256 +Queued experiment '9109ea9' for future execution. ``` -`dvc exp list` shows all experiments that have been saved. +Next, run all (`--run-all`) queued experiments in parallel. You can specify the +number of parallel processes using `--jobs`: ```dvc -$ dvc exp list gitremote --all -72ed9cd: - exp-bfe64 +$ dvc exp run --run-all --jobs 2 ``` -`dvc exp pull` retrieves the experiment from a Git remote. +</details> -```dvc -$ dvc exp pull gitremote exp-bfe64 -Pulled experiment 'exp-bfe64' from Git remote 'gitremote'. -``` +## Comparing and persisting experiments -> All these commands take a Git remote as an argument. A `dvc remote default` is -> also required to share the experiment data. +The experiments are run several times with different parameters. We use +`dvc exp show` to compare all of these experiments. -## Cleaning up +```dvc +$ dvc exp show +``` -Let's take another look at the experiments table: +```dvctable +โโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโณโโโโโโโโโโณโโโโโโโโโณโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโ +โ white:**Experiment** โ white:**Created** โ yellow:**loss** โ yellow:**acc** โ blue:**train.epochs** โ blue:**model.conv_units** โ +โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ +โ workspace โ - โ 0.23508 โ 0.9151 โ 10 โ 24 โ +โ 7317bc6 โ Jul 18, 2021 โ - โ - โ 10 โ 16 โ +โ โโโ e2647ef [exp-ee8a4] โ 05:14 PM โ 0.23146 โ 0.9145 โ 10 โ 64 โ +โ โโโ 15c9451 [exp-a9be6] โ 05:14 PM โ 0.25231 โ 0.9102 โ 10 โ 32 โ +โ โโโ 9c32227 [exp-17dd9] โ 04:46 PM โ 0.23687 โ 0.9167 โ 10 โ 256 โ +โ โโโ 8a9cb15 [exp-29d93] โ 04:46 PM โ 0.24459 โ 0.9134 โ 10 โ 128 โ +โ โโโ dfc536f [exp-a1bd9] โ 03:35 PM โ 0.23508 โ 0.9151 โ 10 โ 24 โ +โ โโโ 1a1d858 [exp-6dccf] โ 03:21 PM โ 0.23282 โ 0.9152 โ 10 โ 16 โ +โโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโดโโโโโโโโโโดโโโโโโโโโดโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโ +``` + +By default, it shows all the parameters and the metrics with the timestamp. If +you have a large number of parameters, metrics or experiments, this may lead to +a cluttered view. You can limit the table to specific metrics, or parameters, or +hide the timestamp column with `--include-metrics`, `--include-params`, or +`--no-timestamp` options of the command, respectively. ```dvc $ dvc exp show --no-timestamp \ - --include-params train.n_est,train.min_split + --include-params model.conv_units --include-metrics acc ``` ```dvctable -โโโโโโโโโโโโโโณโโโโโโโโโโโณโโโโโโโโโโณโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโ -โ neutral:**Experiment** โ metric:**avg_prec** โ metric:**roc_auc** โ param:**train.n_est**โ param:**train.min_split** โ -โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ -โ **workspace** โ **0.60405** โ **0.9608** โ **100** โ **64** โ -โ **master** โ **0.60405** โ **0.9608** โ **100** โ **64** โ -โโโโโโโโโโโโโโดโโโโโโโโโโโดโโโโโโโโโโดโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโ -``` - -Where did all the experiments go? By default, `dvc exp show` only shows -experiments since the last commit, but don't worry. The experiments remain -<abbr>cached</abbr> and can be shown or applied. For example, use `-n` to show -experiments from the previous _n_ commits: +โโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโณโโโโโโโโโโโโโโโโโโโ +โ white:**Experiment** โ yellow:**acc** โ blue:**model.conv_units** โ +โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ +โ workspace โ 0.9151 โ 24 โ +โ 7317bc6 โ - โ 16 โ +โ โโโ e2647ef [exp-ee8a4] โ 0.9145 โ 64 โ +โ โโโ 15c9451 [exp-a9be6] โ 0.9102 โ 32 โ +โ โโโ 9c32227 [exp-17dd9] โ 0.9167 โ 256 โ +โ โโโ 8a9cb15 [exp-29d93] โ 0.9134 โ 128 โ +โ โโโ dfc536f [exp-a1bd9] โ 0.9151 โ 24 โ +โ โโโ 1a1d858 [exp-6dccf] โ 0.9152 โ 16 โ +โโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโดโโโโโโโโโโโโโโโโโโโ +``` + +After selecting an experiment from the table, you can create a Git branch that +contains the experiment with all its related files. ```dvc -$ dvc exp show -n 2 --no-timestamp \ - --include-params train.n_est,train.min_split -``` +$ dvc exp branch exp-05e87 "cnn-256" +Git branch 'cnn-256' has been created from experiment 'exp-05e87'. +To switch to the new branch run: -```dvctable -โโโโโโโโโโโโโโโโโณโโโโโโโโโโโณโโโโโโโโโโณโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโ -โ neutral:**Experiment** โ metric:**avg_prec** โ metric:**roc_auc** โ param:**train.n_est**โ param:**train.min_split** โ -โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ -โ **workspace** โ **0.60405** โ **0.9608** โ **100** โ **64** โ -โ **master** โ **0.60405** โ **0.9608** โ **100** โ **64** โ -โ **64d74b2** โ **0.55259** โ **0.91536** โ **50** โ **2** โ -โ โโโ exp-bfe64 โ 0.57833 โ 0.95555 โ 50 โ 8 โ -โ โโโ exp-b8082 โ 0.59806 โ 0.95287 โ 50 โ 64 โ -โ โโโ exp-c7250 โ 0.58876 โ 0.94524 โ 100 โ 2 โ -โ โโโ exp-98a96 โ 0.60405 โ 0.9608 โ 100 โ 64 โ -โ โโโ exp-b9cd4 โ 0.57953 โ 0.95732 โ 100 โ 8 โ -โ โโโ exp-ad5b1 โ 0.56191 โ 0.93345 โ 50 โ 2 โ -โโโโโโโโโโโโโโโโโดโโโโโโโโโโโดโโโโโโโโโโดโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโ + git checkout cnn-256 ``` -Eventually, old experiments may clutter the experiments table. - -`dvc exp gc` removes all references to old experiments: +You can then checkout and continue working from this branch, or merge the branch +into your `main` branch with the usual Git commands. -```dvc -$ dvc exp gc --workspace -$ dvc exp show -n 2 --no-timestamp \ - --include-params train.n_est,train.min_split -``` +## Go Further -```dvctable -โโโโโโโโโโโโโโณโโโโโโโโโโโณโโโโโโโโโโณโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโ -โ neutral:**Experiment** โ metric:**avg_prec** โ metric:**roc_auc** โ param:**train.n_est**โ param:**train.min_split** โ -โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ -โ **workspace** โ **0.60405** โ **0.9608** โ **100** โ **64** โ -โ **master** โ **0.60405** โ **0.9608** โ **100** โ **64** โ -โ **64d74b2** โ **0.55259** โ **0.91536** โ **50** โ **2** โ -โโโโโโโโโโโโโโดโโโโโโโโโโโดโโโโโโโโโโดโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโ -``` +There are many other features of `dvc exp`, like cleaning up the unused +experiments, sharing them without committing into Git or getting differences +between two experiments. -> `dvc exp gc` only removes references to the experiments; not the cached -> objects associated with them. To clean up the <abbr>cache</abbr>, use -> `dvc gc`. +Please see the section on +[Experiment Management](/doc/user-guide/experiment-management) in the User's +Guide or `dvc exp` and subcommands in the Command Reference.