From 9ec36f2de8db28687a1dde9cef9cd22930c96e19 Mon Sep 17 00:00:00 2001 From: Jorge Orpinel Date: Fri, 10 Dec 2021 10:48:31 -0700 Subject: [PATCH 1/9] ref: bring changes from https://github.com/iterative/dvc.org/pull/2909 --- content/docs/command-reference/exp/apply.md | 26 ++++++----- content/docs/command-reference/exp/branch.md | 20 +++++---- content/docs/command-reference/exp/run.md | 45 +++++++++----------- 3 files changed, 46 insertions(+), 45 deletions(-) diff --git a/content/docs/command-reference/exp/apply.md b/content/docs/command-reference/exp/apply.md index 1bc6b16767..c012e4ef12 100644 --- a/content/docs/command-reference/exp/apply.md +++ b/content/docs/command-reference/exp/apply.md @@ -15,21 +15,25 @@ positional arguments: ## Description Restores an `experiment` into the workspace as long as no more Git commits have -been made after the target experiment (`HEAD` hasn't moved). The `experiment` -can be referenced by name or hash (see `dvc exp run` for details). This changes -any files (code, data, parameters, metrics, etc.) -needed to reflect the experiment conditions and results in the workspace. +been made after the target experiment (`HEAD` hasn't moved). The experiment can +be referenced by name or hash (see `dvc exp run` for details). -⚠️ Conflicting changes in the workspace are overwritten unless `--no-force` is -used. +Specifically, `dvc exp apply` changes any files (code, data, +parameters, metrics, etc.) needed to reflect the +experiment conditions and results in the workspace. Current changes to the +workspace are preserved except if they conflict with the experiment in question. + +⚠️ Conflicting changes in the workspace are overwritten unless unless +`--no-force` is used. This is typically used after choosing a target `experiment` with `dvc exp show` -or `dvc exp diff`, and before committing it to Git (making it -[persistent](/doc/user-guide/experiment-management#persistent-experiments)). +or `dvc exp diff`, and before committing it to Git (making it [persistent]. + +> Note that if a history of [checkpoints] is found in the `experiment`, it will +> **not** be preserved when applying and committing it. -Note that the history of -[checkpoints](/doc/command-reference/exp/run#checkpoints) found in the -`experiment` is **not** preserved when applying and committing it. +[persistent]: /doc/user-guide/experiment-management/persisting-experiments +[checkpoints]: /doc/user-guide/experiment-management/checkpoints ## Options diff --git a/content/docs/command-reference/exp/branch.md b/content/docs/command-reference/exp/branch.md index a4ac9a4c11..50fa193eb9 100644 --- a/content/docs/command-reference/exp/branch.md +++ b/content/docs/command-reference/exp/branch.md @@ -15,26 +15,28 @@ positional arguments: ## Description -Makes a named Git -[`branch`](https://git-scm.com/book/en/v2/Git-Branching-Basic-Branching-and-Merging) -containing the target `experiment` (making it -[persistent](/doc/user-guide/experiment-management#persistent-experiments)). For -[checkpoint experiments](/doc/command-reference/exp/run#checkpoints), the new -branch will contain multiple commits (the checkpoints). +Makes a named Git [`branch`] containing the target `experiment` (making it +[persistent]. For [checkpoint experiments], the new branch will contain multiple +commits (the checkpoints). The new `branch` will be based on the experiment's parent commit (`HEAD` at the time that the experiment was run). Note that DVC **does not** switch into the new `branch` automatically. `dvc exp branch` is useful to make an experiment persistent without modifying -the workspace, so they can be continued, -[stored, and shared](https://dvc.org/doc/use-cases/sharing-data-and-model-files) -in a normal Git + DVC workflow. +the workspace, so they can be continued, [stored and shared] in a normal Git + +DVC workflow. To switch into the new branch, use `git checkout branch` and `dvc checkout`. Or use `git merge branch` and `dvc repro` to combine it with your current project version. +[`branch`]: + https://git-scm.com/book/en/v2/Git-Branching-Basic-Branching-and-Merging +[persistent]: /doc/user-guide/experiment-management/persisting-experiments +[checkpoint experiments]: /doc/command-reference/exp/run#checkpoints +[stored and shared]: /doc/use-cases/sharing-data-and-model-files + ## Options - `-h`, `--help` - shows the help message and exit. diff --git a/content/docs/command-reference/exp/run.md b/content/docs/command-reference/exp/run.md index 869f9a7b8a..37cdc2de8e 100644 --- a/content/docs/command-reference/exp/run.md +++ b/content/docs/command-reference/exp/run.md @@ -22,45 +22,40 @@ Provides a way to execute and track experiments in your project without polluting it with unnecessary commits, branches, directories, etc. -> `dvc exp run` is equivalent to `dvc repro` for experiments. It has the same -> behavior when it comes to `targets` and stage execution (restores the -> dependency graph, etc.). See the command [options](#options) for more on the -> differences. +> `dvc exp run` is equivalent to `dvc repro` for experiments. It +> has the same behavior when it comes to `targets` and stage execution (restores +> the dependency graph, etc.). See the command [options](#options) for more on +> the differences. Before running an experiment, you'll probably want to make modifications such as data and code updates, or hyperparameter tuning. For the latter, you can use the `--set-param` (`-S`) option of this command to change `dvc param` values on-the fly. +📖 See [DVC Experiments](/doc/user-guide/experiment-management) for more +information. + Each experiment creates and tracks a project variation based on your workspace changes. Experiments will have a unique, auto-generated name like `exp-bfe64` by default, which can be customized using the `--name` (`-n`) option. -
- -### ⚙️ How does DVC track experiments? - -Experiments are custom -[Git references](https://git-scm.com/book/en/v2/Git-Internals-Git-References) -(found in `.git/refs/exps`) with a single commit based on `HEAD` (not checked -out by DVC). Note that these commits are not pushed to the Git remote by default -(see `dvc exp push`). +Each experiment creates and tracks a project variation based on the changes in +your workspace. The results of the last `dvc exp run` will be +reflected in the workspace. Experiments will have an auto-generated ID like +`exp-bfe64` by default. A custom name can be given instead, using the `--name` +(`-n`) option -
- -The results of the last `dvc exp run` can be seen in the workspace. To display -and compare multiple experiments, use `dvc exp show` or `dvc exp diff` -(`plots diff` also accepts experiment names as `revisions`). Use `dvc exp apply` -to restore the results of any other experiment instead. - -Successful experiments can be made -[persistent](/doc/user-guide/experiment-management#persistent-experiments) by -committing them to the Git repo. Unnecessary ones can be removed with +To display and compare multiple experiments, use `dvc exp show` or +`dvc exp diff` (`plots diff` also accepts experiment names as `revisions`). Use +`dvc exp apply` to restore the results of any experiment, for example to [commit +them][persisting] to Git. Unnecessary experiments can be removed with `dvc exp remove`or `dvc exp gc` (or abandoned). -> Note that experiment data will remain in the cache until you use -> regular `dvc gc` to clean it up. +> Note that experiment data will remain in the local cache until +> you use regular `dvc gc` to clean it up. + +[persisting]: /doc/user-guide/experiment-management/persisting-experiments ## Checkpoints From d0b46c31e0639ef36422d68f95212debfe945be0 Mon Sep 17 00:00:00 2001 From: Jorge Orpinel Date: Fri, 10 Dec 2021 10:51:37 -0700 Subject: [PATCH 2/9] guide: bring changes from https://github.com/iterative/dvc.org/pull/2909 --- .../user-guide/experiment-management/comparing-experiments.md | 4 ++-- content/docs/user-guide/project-structure/internal-files.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/content/docs/user-guide/experiment-management/comparing-experiments.md b/content/docs/user-guide/experiment-management/comparing-experiments.md index 5974896ad9..a58642e9a3 100644 --- a/content/docs/user-guide/experiment-management/comparing-experiments.md +++ b/content/docs/user-guide/experiment-management/comparing-experiments.md @@ -384,8 +384,8 @@ params.yaml train.epochs 10 10 0 ## Compare an experiment with the workspace When you want to compare two experiments, either the baseline experiment in a -commit, branch, tag or an attached experiment with ID, you can supply their -names to `dvc exp diff`. +commit, branch, or tag; or an attached experiment by name, you can supply any of +these references to `dvc exp diff`. ``` $ dvc exp diff cnn-128 cnn-64 diff --git a/content/docs/user-guide/project-structure/internal-files.md b/content/docs/user-guide/project-structure/internal-files.md index 3a77648119..9339a385a1 100644 --- a/content/docs/user-guide/project-structure/internal-files.md +++ b/content/docs/user-guide/project-structure/internal-files.md @@ -134,7 +134,7 @@ $ cat .dvc/cache/6f/db5336fce0dbfd669f83065f107551.dir That's how DVC knows that the other two cached files belong in the directory. -### Run-cache +## Run-cache `dvc exp run` and `dvc repro` by default populate and reutilize a log of stages that have been run in the project. It is found in the `runs/` directory inside From a029288a45ee064d0449468a0d5cfbd5b6f17e0c Mon Sep 17 00:00:00 2001 From: Jorge Orpinel Date: Tue, 14 Dec 2021 20:43:57 -0700 Subject: [PATCH 3/9] exp: pull some changes from https://github.com/iterative/dvc.org/pull/2908 --- content/docs/command-reference/exp/run.md | 45 ++++++++++--------- content/docs/sidebar.json | 24 +++++----- .../user-guide/experiment-management/index.md | 4 +- content/docs/user-guide/troubleshooting.md | 9 ++-- 4 files changed, 44 insertions(+), 38 deletions(-) diff --git a/content/docs/command-reference/exp/run.md b/content/docs/command-reference/exp/run.md index 37cdc2de8e..4dd24b6896 100644 --- a/content/docs/command-reference/exp/run.md +++ b/content/docs/command-reference/exp/run.md @@ -22,40 +22,45 @@ Provides a way to execute and track experiments in your project without polluting it with unnecessary commits, branches, directories, etc. -> `dvc exp run` is equivalent to `dvc repro` for experiments. It -> has the same behavior when it comes to `targets` and stage execution (restores -> the dependency graph, etc.). See the command [options](#options) for more on -> the differences. +> `dvc exp run` is equivalent to `dvc repro` for experiments. It has the same +> behavior when it comes to `targets` and stage execution (restores the +> dependency graph, etc.). See the command [options](#options) for more on the +> differences. Before running an experiment, you'll probably want to make modifications such as data and code updates, or hyperparameter tuning. For the latter, you can use the `--set-param` (`-S`) option of this command to change `dvc param` values on-the fly. -📖 See [DVC Experiments](/doc/user-guide/experiment-management) for more -information. - Each experiment creates and tracks a project variation based on your workspace changes. Experiments will have a unique, auto-generated name like `exp-bfe64` by default, which can be customized using the `--name` (`-n`) option. -Each experiment creates and tracks a project variation based on the changes in -your workspace. The results of the last `dvc exp run` will be -reflected in the workspace. Experiments will have an auto-generated ID like -`exp-bfe64` by default. A custom name can be given instead, using the `--name` -(`-n`) option +
+ +### ⚙️ How does DVC track experiments? -To display and compare multiple experiments, use `dvc exp show` or -`dvc exp diff` (`plots diff` also accepts experiment names as `revisions`). Use -`dvc exp apply` to restore the results of any experiment, for example to [commit -them][persisting] to Git. Unnecessary experiments can be removed with -`dvc exp remove`or `dvc exp gc` (or abandoned). +Experiments are custom +[Git references](https://git-scm.com/book/en/v2/Git-Internals-Git-References) +(found in `.git/refs/exps`) with a single commit based on `HEAD` (not checked +out by DVC). Note that these commits are not pushed to Git remotes by default +(see `dvc exp push`). -> Note that experiment data will remain in the local cache until -> you use regular `dvc gc` to clean it up. +
+ +The results of the last `dvc exp run` can be seen in the workspace. To display +and compare multiple experiments, use `dvc exp show` or `dvc exp diff` +(`plots diff` also accepts experiment names as `revisions`). Use `dvc exp apply` +to restore the results of any other experiment instead. + +Successful experiments can be made +[persistent](/doc/user-guide/experiment-management#persistent-experiments) by +committing them to the Git repo. Unnecessary ones can be removed with +`dvc exp remove`or `dvc exp gc` (or abandoned). -[persisting]: /doc/user-guide/experiment-management/persisting-experiments +> Note that experiment data will remain in the cache until you use +> regular `dvc gc` to clean it up. ## Checkpoints diff --git a/content/docs/sidebar.json b/content/docs/sidebar.json index f68a665d58..65fb2e7117 100644 --- a/content/docs/sidebar.json +++ b/content/docs/sidebar.json @@ -132,18 +132,6 @@ } ] }, - { - "label": "How To", - "slug": "how-to", - "source": false, - "children": [ - "stop-tracking-data", - "update-tracked-data", - "add-deps-or-outs-to-a-stage", - "merge-conflicts", - "share-a-dvc-cache" - ] - }, { "label": "Experiment Management", "slug": "experiment-management", @@ -158,6 +146,18 @@ "checkpoints" ] }, + { + "slug": "how-to", + "source": false, + "children": [ + "stop-tracking-data", + "update-tracked-data", + "add-deps-or-outs-to-a-stage", + "merge-conflicts", + "share-a-dvc-cache", + "share-many-experiments" + ] + }, "setup-google-drive-remote", "large-dataset-optimization", "external-dependencies", diff --git a/content/docs/user-guide/experiment-management/index.md b/content/docs/user-guide/experiment-management/index.md index b501fd0a16..b032c7e18e 100644 --- a/content/docs/user-guide/experiment-management/index.md +++ b/content/docs/user-guide/experiment-management/index.md @@ -47,8 +47,8 @@ main alternatives: - **Git tags and branches** - use the repo's "time dimension" to distribute your experiments. This makes the most sense for experiments that build on each - other. Helpful if the Git [revisions](https://git-scm.com/docs/revisions) can - be easily visualized, for example with tools + other. Git-based experiment structures are especially helpful along with Git + history exploration tools [like GitHub](https://docs.github.com/en/github/visualizing-repository-data-with-graphs/viewing-a-repositorys-network). - **Directories** - the project's "space dimension" can be structured with diff --git a/content/docs/user-guide/troubleshooting.md b/content/docs/user-guide/troubleshooting.md index 4804adb508..2eef09b77f 100644 --- a/content/docs/user-guide/troubleshooting.md +++ b/content/docs/user-guide/troubleshooting.md @@ -92,16 +92,17 @@ using: $ dvc checkout --relink ``` -## HTTP Git authentication is not supported {#git-auth} +## DVC can only authenticate with Git remotes using SSH URLs {#git-auth} [Experiment sharing](/doc/user-guide/experiment-management/sharing-experiments) -commands accept a `git_remote` argument. In order to access the Git remote, you -may need to authenticate for _write_ (`dvc exp push`) or _read_ (`dvc exp list`, +commands accept a `git_remote` argument. You may need to authenticate to use the +Git remote, for _write_ (`dvc exp push`) or _read_ (`dvc exp list`, `dvc exp pull`) permissions. DVC does not currently support authentication with [Git credentials]. This means that unless the Git server allows unauthenticated HTTP write/read, you should -use an [SSH Git URL] when listing, pulling or pushing experiments. +use an [SSH Git URL] for Git remotes used for listing, pulling or pushing +experiments. [git credentials]: https://git-scm.com/docs/gitcredentials [ssh git url]: From 49c585cd69eb69da0befaeb61435fb14504a2b2e Mon Sep 17 00:00:00 2001 From: Jorge Orpinel Date: Tue, 21 Dec 2021 00:51:10 -0700 Subject: [PATCH 4/9] ref: better explain exp apply behavior explicitly rel. https://github.com/iterative/dvc.org/pull/3080#discussion_r769709333 --- content/docs/command-reference/exp/apply.md | 21 ++++++++++++--------- 1 file changed, 12 insertions(+), 9 deletions(-) diff --git a/content/docs/command-reference/exp/apply.md b/content/docs/command-reference/exp/apply.md index c012e4ef12..7fc1d72597 100644 --- a/content/docs/command-reference/exp/apply.md +++ b/content/docs/command-reference/exp/apply.md @@ -14,17 +14,20 @@ positional arguments: ## Description -Restores an `experiment` into the workspace as long as no more Git commits have -been made after the target experiment (`HEAD` hasn't moved). The experiment can -be referenced by name or hash (see `dvc exp run` for details). +Restores an `experiment` into the workspace, as long as we're on the same +project baseline (Git `HEAD`) as when the target experiment was run. The +experiment can be referenced by name or hash (see `dvc exp run` for details). -Specifically, `dvc exp apply` changes any files (code, data, -parameters, metrics, etc.) needed to reflect the -experiment conditions and results in the workspace. Current changes to the -workspace are preserved except if they conflict with the experiment in question. +Specifically, `dvc exp apply` checks out any files or directories needed to +reflect the experiment conditions and results. This can include both with DVC +and Git: code, data, parameters, metrics, etc. -⚠️ Conflicting changes in the workspace are overwritten unless unless -`--no-force` is used. +> Note that existing changes in the workspace that don't conflict the +> `experiment` are left alone. This may result in a mix of other changes with +> the ones applied by this command. + +⚠️ Conflicting changes in the workspace are overwritten unless `--no-force` is +used. This is typically used after choosing a target `experiment` with `dvc exp show` or `dvc exp diff`, and before committing it to Git (making it [persistent]. From 220a9b002b1328488ad8bf1ae8564c30ccedbf14 Mon Sep 17 00:00:00 2001 From: Jorge Orpinel Date: Tue, 21 Dec 2021 01:24:22 -0700 Subject: [PATCH 5/9] ref: exp apply destroys any canges per https://github.com/iterative/dvc.org/pull/3080#discussion_r772903011 --- content/docs/command-reference/exp/apply.md | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/content/docs/command-reference/exp/apply.md b/content/docs/command-reference/exp/apply.md index 7fc1d72597..1153012e6f 100644 --- a/content/docs/command-reference/exp/apply.md +++ b/content/docs/command-reference/exp/apply.md @@ -22,12 +22,8 @@ Specifically, `dvc exp apply` checks out any files or directories needed to reflect the experiment conditions and results. This can include both with DVC and Git: code, data, parameters, metrics, etc. -> Note that existing changes in the workspace that don't conflict the -> `experiment` are left alone. This may result in a mix of other changes with -> the ones applied by this command. - -⚠️ Conflicting changes in the workspace are overwritten unless `--no-force` is -used. +⚠️ This command will destroy any existing changes in the workspace (Git working +tree) unless the `--no-force` flag is used. This is typically used after choosing a target `experiment` with `dvc exp show` or `dvc exp diff`, and before committing it to Git (making it [persistent]. From 7c30e282390865b1863d9939eb42375b5b437fed Mon Sep 17 00:00:00 2001 From: Jorge Orpinel Date: Tue, 21 Dec 2021 01:32:48 -0700 Subject: [PATCH 6/9] ref: clarify that exp branch makes 1+ commits in the exp branch rel. https://github.com/iterative/dvc.org/pull/3080#discussion_r769714007 --- content/docs/command-reference/exp/branch.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/content/docs/command-reference/exp/branch.md b/content/docs/command-reference/exp/branch.md index 50fa193eb9..cdbc88f747 100644 --- a/content/docs/command-reference/exp/branch.md +++ b/content/docs/command-reference/exp/branch.md @@ -15,9 +15,9 @@ positional arguments: ## Description -Makes a named Git [`branch`] containing the target `experiment` (making it -[persistent]. For [checkpoint experiments], the new branch will contain multiple -commits (the checkpoints). +Makes a given Git [`branch`] containing the target `experiment`. This makes the +experiment into a [regular commit], or several in the case of [checkpoint +experiments] (one per checkpoint). The new `branch` will be based on the experiment's parent commit (`HEAD` at the time that the experiment was run). Note that DVC **does not** switch into the @@ -33,7 +33,7 @@ version. [`branch`]: https://git-scm.com/book/en/v2/Git-Branching-Basic-Branching-and-Merging -[persistent]: /doc/user-guide/experiment-management/persisting-experiments +[regular commit]: /doc/user-guide/experiment-management/persisting-experiments [checkpoint experiments]: /doc/command-reference/exp/run#checkpoints [stored and shared]: /doc/use-cases/sharing-data-and-model-files From de850f5d59030e6c7fd5feb7c439f81e7fe2c594 Mon Sep 17 00:00:00 2001 From: Jorge Orpinel Date: Wed, 22 Dec 2021 22:58:14 -0700 Subject: [PATCH 7/9] ref: link exp apply -> exp branch per https://github.com/iterative/dvc.org/pull/3080#discussion_r774052629 --- content/docs/command-reference/exp/apply.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/docs/command-reference/exp/apply.md b/content/docs/command-reference/exp/apply.md index 1153012e6f..44a957013c 100644 --- a/content/docs/command-reference/exp/apply.md +++ b/content/docs/command-reference/exp/apply.md @@ -28,8 +28,8 @@ tree) unless the `--no-force` flag is used. This is typically used after choosing a target `experiment` with `dvc exp show` or `dvc exp diff`, and before committing it to Git (making it [persistent]. -> Note that if a history of [checkpoints] is found in the `experiment`, it will -> **not** be preserved when applying and committing it. +> Note that any [checkpoints] found in the `experiment` will **not** be +> preserved when applying and committing it. Use `dvc exp branch` instead. [persistent]: /doc/user-guide/experiment-management/persisting-experiments [checkpoints]: /doc/user-guide/experiment-management/checkpoints From 817d6eeeda3781cfa344d534fcb60c756f82ea50 Mon Sep 17 00:00:00 2001 From: Jorge Orpinel Date: Thu, 23 Dec 2021 11:15:03 -0700 Subject: [PATCH 8/9] ref: clarify (again) what applies does specifically per https://github.com/iterative/dvc.org/pull/3080#pullrequestreview-839250938 --- content/docs/command-reference/exp/apply.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/content/docs/command-reference/exp/apply.md b/content/docs/command-reference/exp/apply.md index 44a957013c..8b7a9a5a9c 100644 --- a/content/docs/command-reference/exp/apply.md +++ b/content/docs/command-reference/exp/apply.md @@ -18,9 +18,10 @@ Restores an `experiment` into the workspace, as long as we're on the same project baseline (Git `HEAD`) as when the target experiment was run. The experiment can be referenced by name or hash (see `dvc exp run` for details). -Specifically, `dvc exp apply` checks out any files or directories needed to -reflect the experiment conditions and results. This can include both with DVC -and Git: code, data, parameters, metrics, etc. +Specifically, `dvc exp apply` restores any files or directories needed to +reflect the experiment conditions and results. This means checking out files +tracked both with DVC and Git: code, raw data, parameters, +metrics, resulting artifacts, etc. ⚠️ This command will destroy any existing changes in the workspace (Git working tree) unless the `--no-force` flag is used. From b2bcddc3f292006924d55d5e1c722d559a89134c Mon Sep 17 00:00:00 2001 From: Jorge Orpinel Date: Thu, 23 Dec 2021 11:19:59 -0700 Subject: [PATCH 9/9] ref: document intended apply behavior on conflicting changes per https://github.com/iterative/dvc.org/pull/3080#pullrequestreview-838685836 --- content/docs/command-reference/exp/apply.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/docs/command-reference/exp/apply.md b/content/docs/command-reference/exp/apply.md index 8b7a9a5a9c..e482aa4b3f 100644 --- a/content/docs/command-reference/exp/apply.md +++ b/content/docs/command-reference/exp/apply.md @@ -23,8 +23,8 @@ reflect the experiment conditions and results. This means checking out files tracked both with DVC and Git: code, raw data, parameters, metrics, resulting artifacts, etc. -⚠️ This command will destroy any existing changes in the workspace (Git working -tree) unless the `--no-force` flag is used. +⚠️ Conflicting changes in the workspace are overwritten unless `--no-force` is +used. This is typically used after choosing a target `experiment` with `dvc exp show` or `dvc exp diff`, and before committing it to Git (making it [persistent].