guide: ML Pipelines (1): Defining Pipelines & Stages

content/docs/command-reference/dag.md

@@ -1,7 +1,7 @@
 # dag
 
-Visualize the <abbr>pipeline</abbr>(s) in `dvc.yaml` as one or more graph(s) of
-connected [stages](/doc/command-reference/run).
+Visualize <abbr>pipelines</abbr> as one or more <abbr>stage</abbr> dependency
+graphs.
 
 ## Synopsis
 
@@ -17,28 +17,15 @@ positional arguments:
 
 ## Description
 
-Displays the stages of a pipeline up to the `target` stage. If the `target` is
-omitted, it will show the full project DAG.
+DVC represents a pipeline internally as a **Directed Acyclic Graph** (DAG) where
+the nodes are stages and the edges are dependencies.
 
-### Directed acyclic graph
+`dvc dag` displays this dependency graph in one or more pipelines, as defined in
+the `dvc.yaml` files found in the <abbr>project</abbr>. Provide a `target` stage
+name to show the pipeline up to that point.
 
-A data pipeline, in general, is a series of data processing
-[stages](/doc/command-reference/run) (for example, console commands that take an
-input and produce an outcome). The connections between stages are formed by the
-<abbr>output</abbr> of one turning into the <abbr>dependency</abbr> of another.
-A pipeline may produce intermediate data, and has a final result.
-
-Data science and machine learning pipelines typically start with large raw
-datasets, include intermediate featurization and training stages, and produce a
-final model, as well as accuracy [metrics](/doc/command-reference/metrics).
-
-In DVC, pipeline stages and commands, their data I/O, interdependencies, and
-results (intermediate or final) are specified in `dvc.yaml`, which can be
-written manually or built using the helper command `dvc stage add`. This allows
-DVC to restore one or more pipelines later (see `dvc repro`).
-
-> DVC builds a dependency graph
-> ([DAG](https://en.wikipedia.org/wiki/Directed_acyclic_graph)) to do this.
+[directed acyclic graph]:
+  /doc/user-guide/data-pipelines/defining-pipelines#directed-acyclic-graph-dag
 
 ### Paginating the output
 

content/docs/command-reference/exp/init.md

@@ -14,77 +14,90 @@ usage: dvc exp init [-h] [-q | -v] [--run] [--interactive] [-f]
                     [--metrics METRICS] [--plots PLOTS] [--live LIVE]
                     [--type {default,checkpoint}]
                     [command]
+
+positional arguments:
+  command               Shell command to runs the experiment(s)
 ```
 
 ## Description
 
 This command helps you get started with DVC Experiments quickly. It reduces
-repetitive DVC procedures by creating a necessary `dvc.yaml` file, which assumes
-standard locations of your inputs (data, <abbr>parameters</abbr>, and source
-code) and outputs (models, <abbr>metrics</abbr>, and
+repetitive DVC procedures by creating a `dvc.yaml` file. It assumes standard
+locations of your inputs (data, <abbr>parameters</abbr>, and source code) and
+outputs (models, <abbr>metrics</abbr>, and
 [plots](/doc/command-reference/plots)).
 
-These locations can be customized through the [command options](#options) or via
-[configuration](/doc/command-reference/config#exp). Default project structure:
+The only required argument is a [shell `command`] to run your experiment(s). It
+can be provided directly as an argument (see example below) or by using the
+`--interactive` (`-i`) mode, which will prompt for it.
+
+```cli
+$ dvc exp init "python src/train.py"
+Creating dependencies: src, data and params.yaml
+Creating output directories: plots and models
+Creating train stage in dvc.yaml
+```
+
+`dvc exp init` also generates the boilerplate project structure, including input
+files/directories and directories needed for future outputs. These locations can
+also be customized via [CLI options](#options) or interactive mode, or with
+[configuration](/doc/command-reference/config#exp). Default structure:
 
 ```
 ├── data/
+├── dvc.yaml
 ├── metrics.json
 ├── models/
 ├── params.yaml
 ├── plots/
 └── src/
 ```
 
-The only required argument is the terminal command that runs your experiment(s).
-It can be provided directly [as an argument](#the-command-argument) or by using
-the `--interactive` (`-i`) mode (which will prompt for it). The command will be
-wrapped as a <abbr>stage</abbr> that `dvc exp run` can execute.
+Inside `dvc.yaml`, the experiment is wrapped as a <abbr>stage</abbr> that
+`dvc exp run` can execute.
 
-<admon type="tip">
+<details>
 
-A special `--type` of stage is supported (`checkpoint`), which monitors
-[checkpoints] during training of ML models.
+### Click to see `dvc.yaml` example
 
-</admon>
+```yaml
+stages:
+  train:
+    cmd: python src/train.py
+    deps:
+      - data
+      - src
+    params:
+      - params.yaml:
+    outs:
+      - models
+    metrics:
+      - metrics.json:
+          cache: false
+    plots:
+      - plots:
+          cache: false
+```
 
-`dvc exp init` also generates the boilerplate project structure, including input
-files/directories and directories needed for future outputs, or any locations
-determined in interactive mode.
+</details>
 
-<admon type="info">
+<admon type="tip">
 
-`dvc exp init` is intended as a quick way to start running [DVC Experiments].
-See the `dvc.yaml` specification for more complex data pipelines.
+A special `--type` of stage is supported (`checkpoint`), which monitors
+[checkpoints] during training of ML models.
 
 </admon>
 
+📖 `dvc exp init` is intended as a quick way to start running [DVC Experiments].
+See the [Pipelines guide] for more on that topic.
+
 [stage definition]:
   /doc/user-guide/project-structure/dvcyaml-files#stage-entries
+[shell `command`]:
+  /doc/user-guide/project-structure/dvcyaml-files#stage-commands
 [checkpoints]: /doc/user-guide/experiment-management/checkpoints
 [dvc experiments]: /doc/user-guide/experiment-management/experiments-overview
-
-### The `command` argument
-
-The command given to `dvc exp init` can be anything your system terminal would
-accept and run directly, for example a shell built-in, an expression, or a
-binary found in `PATH`. Please note that any flags sent after the `command`
-argument will normally become part of that command itself and ignored by
-`dvc exp init` (so provide it last).
-
-⚠️ While DVC is platform-agnostic, commands defined in `dvc.yaml` (`cmd` field)
-may only work on some operating systems and require certain software packages or
-libraries in the environment.
-
-Surround the command with double quotes `"` if it includes special characters
-like `|` or `<`, `>` -- otherwise they would apply to `dvc exp init` itself. Use
-single quotes `'` instead if there are environment variables in it that should
-be evaluated dynamically.
-
-```dvc
-$ dvc exp init "./a_script.sh > /dev/null 2>&1"
-$ dvc exp init './another_script.sh $MYENVVAR'
-```
+[pipelines guide]: /doc/user-guide/data-pipelines/defining-pipelines
 
 ## Options
 

content/docs/command-reference/move.md

@@ -87,11 +87,15 @@ model file:
 $ mv keras.h5 model.h5
 ```
 
-> Note that, often the output of a stage is a dependency in another stage,
-> creating a
-> [dependency graph](/doc/command-reference/run#dependencies-and-outputs). In
-> this case, you may want to also update the `path` in the `deps` field of
-> `dvc.yaml`.
+<admon type="info">
+
+Often the output of a stage is a dependency in another stage, creating a
+[dependency graph]. In this case, you may want to also update the `path` in the
+`deps` field of `dvc.yaml`.
+
+[dependency graph]: /doc/user-guide/data-pipelines/defining-pipelines
+
+</admon>
 
 Finally, we run `dvc commit` with the `-f` option to force save the changes to
 <abbr>cache</abbr>:

content/docs/command-reference/repro.md

@@ -22,10 +22,9 @@ positional arguments:
 
 ## Description
 
-Provides a way to regenerate data pipeline results, by restoring the dependency
-graph (a [DAG](https://en.wikipedia.org/wiki/Directed_acyclic_graph)) implicitly
-defined by the stages listed in `dvc.yaml`. The commands defined in these stages
-are then executed in the correct order.
+Provides a way to regenerate data pipeline results, by restoring the [dependency
+graph] implicitly defined by the stages listed in `dvc.yaml`. The commands
+defined in these stages are then executed in the correct order.
 
 For stages with multiple commands (having a list in the `cmd` field), commands
 are run one after the other in the order they are defined. The failure of any
@@ -69,6 +68,7 @@ It stores all the data files, intermediate or final results in the
 hash values of changed dependencies and outputs in the `dvc.lock` and `.dvc`
 files.
 
+[dependency graph]: /doc/user-guide/data-pipelines/defining-pipelines
 [always changed]: /doc/command-reference/status#local-workspace-status
 
 ### Parallel stage execution
@@ -163,7 +163,7 @@ up-to-date and only execute the final stage.
   with the same dependencies and outputs (see the
   [details](/doc/user-guide/project-structure/internal-files#run-cache)). Useful
   for example if the stage command/s is/are non-deterministic
-  ([not recommended](/doc/command-reference/run#avoiding-unexpected-behavior)).
+  ([not recommended](/doc/user-guide/data-pipelines/defining-pipelines#avoiding-unexpected-behavior)).
 
 - `--force-downstream` - in cases like `... -> A (changed) -> B -> C` it will
   reproduce `A` first and then `B`, even if `B` was previously executed with the

content/docs/command-reference/run.md

@@ -34,44 +34,29 @@ organize data science projects, or build detailed machine learning pipelines.
 A stage name is required and can be provided using the `-n` (`--name`) option.
 The other available [options](#options) are mostly meant to describe different
 kinds of stage [dependencies and outputs](#dependencies-and-outputs). The
-remaining terminal input provided to `dvc run` after `-`/`--` flags will become
-the required [`command` argument](#the-command-argument).
+remaining terminal input provided to `dvc run` after any options/flags will
+become the required [`command` argument].
 
-`dvc run` executes stage commands, unless the `--no-exec` option is used.
-
-<details>
-
-### 💡 Avoiding unexpected behavior
-
-We don't want to tell anyone how to write their code or what programs to use!
-However, please be aware that in order to prevent unexpected results when DVC
-reproduces pipeline stages, the underlying code should ideally follow these
-rules:
-
-- Read/write exclusively from/to the specified <abbr>dependencies</abbr> and
-  <abbr>outputs</abbr> (including parameters files, metrics, and plots).
+<admon type="info">
 
-- Completely rewrite outputs. Do not append or edit.
+`-`/`--` flags sent after the `command` become part of the command itself and
+are ignored by `dvc stage add`.
 
-- Stop reading and writing files when the `command` exits.
+</admon>
 
-Also, if your pipeline reproducibility goals include consistent output data, its
-code should be
-[deterministic](https://en.wikipedia.org/wiki/Deterministic_algorithm) (produce
-the same output for any given input): avoid code that increases
-[entropy](https://en.wikipedia.org/wiki/Software_entropy) (e.g. random numbers,
-time functions, hardware dependencies, etc.).
+`dvc run` executes stage commands, unless the `--no-exec` option is used.
 
-</details>
+[`command` argument]:
+  /doc/user-guide/project-structure/dvcyaml-files#stage-commands
 
 ### Dependencies and outputs
 
 By specifying lists of <abbr>dependencies</abbr> (`-d` option) and/or
 <abbr>outputs</abbr> (`-o` and `-O` options) for each stage, we can create a
-_dependency graph_ ([DAG](https://en.wikipedia.org/wiki/Directed_acyclic_graph))
-that connects them, i.e. the output of a stage becomes the input of another, and
-so on (see `dvc dag`). This graph can be restored by DVC later to modify or
-[reproduce](/doc/command-reference/repro) the full pipeline. For example:
+[dependency graph] that connects them, i.e. the output of a stage becomes the
+input of another, and so on (see `dvc dag`). This graph can be restored by DVC
+later to modify or [reproduce](/doc/command-reference/repro) the full pipeline.
+For example:
 
 ```dvc
 $ dvc run -n printer -d write.sh -o pages ./write.sh
@@ -122,39 +107,23 @@ Relevant notes:
   [manual process](/doc/command-reference/move#renaming-stage-outputs) to update
   `dvc.yaml` and the project's cache accordingly.
 
+[dependency graph]: /doc/user-guide/data-pipelines/defining-pipelines
+
 ### For displaying and comparing data science experiments
 
-[parameters](/doc/command-reference/params) (`-p`/`--params` option) are a
-special type of key/value dependencies. Multiple parameter dependencies can be
-specified from within one or more YAML, JSON, TOML, or Python parameters files
-(e.g. `params.yaml`). This allows tracking experimental hyperparameters easily.
+[parameters][param-deps] (`-p`/`--params` option) are a special type of
+key/value dependencies. Multiple params can be specified from within one or more
+structured files (`params.yaml` by default). This allows tracking experimental
+hyperparameters easily in ML.
 
 Special types of output files, [metrics](/doc/command-reference/metrics) (`-m`
 and `-M` options) and [plots](/doc/command-reference/plots) (`--plots` and
 `--plots-no-cache` options), are also supported. Metrics and plots files have
 specific formats (JSON, YAML, CSV, or TSV) and allow displaying and comparing
 data science experiments.
 
-### The `command` argument
-
-The `command` sent to `dvc run` can be anything your terminal would accept and
-run directly, for example a shell built-in, expression, or binary found in
-`PATH`. Please remember that any flags sent after the `command` are interpreted
-by the command itself, not by `dvc run`.
-
-⚠️ While DVC is platform-agnostic, the commands defined in your
-[pipeline](/doc/command-reference/dag) stages may only work on some operating
-systems and require certain software packages to be installed.
-
-Wrap the command with double quotes `"` if there are special characters in it
-like `|` (pipe) or `<`, `>` (redirection), otherwise they would apply to
-`dvc run` itself. Use single quotes `'` instead if there are environment
-variables in it that should be evaluated dynamically. Examples:
-
-```dvc
-$ dvc run -n first_stage "./a_script.sh > /dev/null 2>&1"
-$ dvc run -n second_stage './another_script.sh $MYENVVAR'
-```
+[param-deps]:
+  /doc/user-guide/pipelines/defining-pipelines#parameter-dependencies
 
 ## Options
 
@@ -199,12 +168,12 @@ $ dvc run -n second_stage './another_script.sh $MYENVVAR'
   makes the stage incompatible with `dvc repro`. Implies `--no-exec`.
 
 - `-p [<path>:]<params_list>`, `--params [<path>:]<params_list>` - specify one
-  or more [parameter dependencies](/doc/command-reference/params) from a
-  parameters file `path` (`./params.yaml` by default). This is done by sending a
-  comma separated list (`params_list`) as argument, e.g.
-  `-p learning_rate,epochs`. A custom params file can be defined with a prefix,
-  e.g. `-p params.json:threshold`. Or use the prefix alone with `:` to use all
-  the parameters found in that file, e.g. `-p myparams.toml:`.
+  or more [parameter dependencies][param-deps] from a structured file `path`
+  (`./params.yaml` by default). This is done by sending a comma separated list
+  (`params_list`) as argument, e.g. `-p learning_rate,epochs`. A custom params
+  file can be defined with a prefix, e.g. `-p params.json:threshold`. Or use the
+  prefix alone with `:` to use all the parameters found in that file, e.g.
+  `-p myparams.toml:`.
 
 - `-m <path>`, `--metrics <path>` - specify a metrics file produced by this
   stage. This option behaves like `-o` but registers the file in a `metrics`
@@ -250,7 +219,7 @@ $ dvc run -n second_stage './another_script.sh $MYENVVAR'
   run with the same dependencies and outputs (see the
   [details](/doc/user-guide/project-structure/internal-files#run-cache)). Useful
   for example if the stage command/s is/are non-deterministic
-  ([not recommended](#avoiding-unexpected-behavior)).
+  ([not recommended](/doc/user-guide/data-pipelines/defining-pipelines#avoiding-unexpected-behavior)).
 
 - `--no-commit` - do not store the outputs of this execution in the cache
   (`dvc.yaml` and `dvc.lock` are still created or updated); useful to avoid