diff --git a/content/docs/command-reference/add.md b/content/docs/command-reference/add.md index 9b4e2610d2..c5766a4074 100644 --- a/content/docs/command-reference/add.md +++ b/content/docs/command-reference/add.md @@ -58,7 +58,7 @@ other DVC commands), a few actions are taken under the hood: [remote storage]: /doc/user-guide/data-management/remote-storage [structure of cache directory]: /doc/user-guide/project-structure/internal-files#structure-of-the-cache-directory -[`core.autostage`]: /doc/command-reference/config#core +[`core.autostage`]: /doc/user-guide/project-structure/configuration#core Summarizing, the result is that the target data is replaced by small `.dvc` files that can be easily tracked with Git. diff --git a/content/docs/command-reference/checkout.md b/content/docs/command-reference/checkout.md index a1d3e419e1..f7473da9d2 100644 --- a/content/docs/command-reference/checkout.md +++ b/content/docs/command-reference/checkout.md @@ -45,15 +45,18 @@ after `git checkout`. See the for more details. By default, this command tries not make copies of cached files in the workspace, -using reflinks instead when supported by the file system (refer to -[File link types](/doc/user-guide/data-management/large-dataset-optimization#file-link-types-for-the-dvc-cache)). -The next linking strategy default value is `copy` though, so unless other file -link types are manually configured in `cache.type` (using `dvc config`), files -will be copied. Keep in mind that having file copies doesn't present much of a -negative impact unless the project uses very large data (several GBs or more). -But leveraging file links is crucial with large files, for example when checking -out a 50Gb file by copying might take a few minutes whereas, with links, -restoring any file size will be almost instantaneous. +using reflinks instead when supported by the file system (refer to [File link +types]). The next linking strategy default value is `copy` though, so unless +other file link types are manually configured in [`cache.type`]), files will be +copied. Keep in mind that having file copies doesn't present much of a negative +impact unless the project uses very large data (several GBs or more). But +leveraging file links is crucial with large files, for example when checking out +a 50Gb file by copying might take a few minutes whereas, with links, restoring +any file size will be almost instantaneous. + +[File link types]: + /doc/user-guide/data-management/large-dataset-optimization#file-link-types-for-the-dvc-cache +[`cache.type`]: /doc/user-guide/project-structure/configuration#cache > When linking files takes longer than expected (10 seconds for any one file) > and `cache.type` is not set, a warning will be displayed reminding users about @@ -95,7 +98,8 @@ situation. In some cases, the data can be pulled from [remote storage] using - `--relink` - ensures the file linking strategy (`reflink`, `hardlink`, `symlink`, or `copy`) for all data in the workspace is consistent with the - project's [`cache.type`](/doc/command-reference/config#cache). This is + project's + [`cache.type`](/doc/user-guide/project-structure/configuration#cache). This is achieved by restoring **all data files or directories** referenced in current DVC files (regardless of whether the files/dirs were already present). diff --git a/content/docs/command-reference/config.md b/content/docs/command-reference/config.md index a6c8b64ae1..79b9acfc36 100644 --- a/content/docs/command-reference/config.md +++ b/content/docs/command-reference/config.md @@ -17,9 +17,9 @@ positional arguments: ## Description -You can query/set/replace/unset DVC configuration options with this command. It -takes a config option `name` (a config section and a key, separated by a dot) -and its `value` (any valid alpha-numeric string generally). +You can query/set/replace/unset [DVC configuration] options with this command. +It takes a config option `name` (a [config section] and a key, separated by a +dot) and its `value` (any valid alpha-numeric string generally). When reading config options (no `value` is given or `--list` is used), the values are read from a combined set of values from the system, global, project, @@ -30,6 +30,10 @@ When writing (a `value` is given or `--unset` is used), the new value is written to the project-level config file by default (`.dvc/config`). Options `--system`, `--global` and `--local` can be used to write to that location instead. +[dvc configuration]: /doc/user-guide/project-structure/configuration +[config section]: + /doc/user-guide/project-structure/configuration#configuration-sections + `.dvc/config` is meant to be tracked by Git and should not contain sensitive @@ -57,11 +61,22 @@ multiple projects or users, respectively. | `--global` | 3 | `$HOME/Library/Application\ Support/dvc/config` | `$HOME/.config/dvc/config` | `%LocalAppData%\iterative\dvc\config` | | `--system` | 4 | `/Library/Application\ Support/dvc/config` | `/etc/xdg/dvc/config` | `%AllUsersProfile%\Application Data\iterative\dvc\config` | -> \* For Linux, the global `dvc/config` may be found in `$XDG_CONFIG_HOME`, and -> the system-wide one in `$XDG_CONFIG_DIRS[0]`, if those env vars are defined. + + +\* For Linux, the global file may be found in `$XDG_CONFIG_HOME`, and the system +file in `$XDG_CONFIG_DIRS[0]`, if those env vars are defined. + + + + + +The `--show-origin` flag can show you where a given config option `value` is +currently stored. + + -> Note that the `--show-origin` flag can show you where a given config option -> `value` is currently stored. +Refer to [Configuration sections][config section] to explore which config +options (`name`) are accepted. ## Command options/flags @@ -97,333 +112,6 @@ multiple projects or users, respectively. - `-v`, `--verbose` - displays detailed tracing information. -## Configuration sections - -The following config sections are written by this command to the appropriate -config file (`.dvc/config` by default), supporting different config options -within: - -- [`core`](#core) - main section with the general config options -- [`remote`](#remote) - sections in the config file that describe particular - remotes -- [`cache`](#cache) - options that affect the project's cache -- [`exp`](#exp) - options to change the default repo paths assumed by - `dvc exp init` -- [`hydra`](#hydra) - options around [Hydra Composition] for experiment - configuration. -- [`parsing`](#parsing) - options around the parsing of [dictionary unpacking]. -- [`plots`](#plots) - options for configuring `dvc plots`. -- [`state`](#state) - see [Internal directories and files][internals] to learn - more about the state database. -- [`index`](#index) - see [Internal directories and files][internals] to learn - more about remote index files. - -[hydra composition]: /doc/user-guide/experiment-management/hydra-composition -[dictionary unpacking]: - /doc/user-guide/project-structure/dvcyaml-files#dictionary-unpacking -[internals]: /doc/user-guide/project-structure/internal-files - -### core - -- `core.remote` - name of the [remote storage](#remote) to use by default. - -- `core.interactive` - whether to always ask for confirmation before reproducing - each [stage](/doc/command-reference/run) in `dvc repro`. (Normally, this - behavior requires using the `-i` option of that command.) Accepts values: - `true` and `false`. - -- `core.analytics` - used to turn off - [anonymized usage statistics](/doc/user-guide/analytics). Accepts values - `true` (default) and `false`. - -- `core.checksum_jobs` - number of threads for computing file hashes. Accepts - positive integers. The default value is `max(1, min(4, cpu_count() // 2))`. - -- `core.hardlink_lock` - use hardlink file locks instead of the default ones, - based on [`flock`](https://linux.die.net/man/2/flock) (i.e. project lock file - `.dvc/lock`). Accepts values `true` and `false` (default). Useful when the DVC - project is on a file system that doesn't properly support file locking (e.g. - [NFS v3 and older](http://nfs.sourceforge.net/)). - -- `core.no_scm` - tells DVC to not expect or integrate with Git (even if the - project is initialized inside a Git repo). Accepts values `true` - and `false` (default). Set with the `--no-scm` option of `dvc init` - ([more details](/doc/command-reference/init#initializing-dvc-without-git)). - -- `core.check_update` - disable/enable DVC's automatic update checks, which - notify the user when a new version is available. Accepts values `true` - (default) and `false`. - -- `core.autostage` - if enabled, DVC will automatically stage (`git add`) - DVC files created or modified by DVC commands. The files will not - be committed. Accepts values `true` and `false` (default). - -### remote - -Unlike most other sections, configuration files may have more than one -`'remote'`. All of them require a unique `"name"` and a `url` value. They can -also specify `jobs`, `verify`, and many platform-specific key/value pairs like -`port` and `password`. - - - -See [Remote Storage Configuration] for more details. - -[remote storage configuration]: - /doc/user-guide/data-management/remote-storage#configuration - - - -For example, the following config file defines a `temp` remote in the local file -system (located in `/tmp/dvcstore`), and marked as default (via [`core`](#core) -section): - -```ini -['remote "temp"'] - url = /tmp/dvcstore -[core] - remote = temp -``` - -### cache - -- `cache.dir` - set/unset cache directory location. A correct value is either an - absolute path, or a path **relative to the config file location**. The default - value is `cache`, that resolves to `.dvc/cache` (relative to the project - config file location). - - - - See also the helper command `dvc cache dir` to intuitively set this config - option, properly transforming paths relative to the current working directory - into paths relative to the config file location. - - - -- `cache.type` - link type that DVC should use to link data files from cache to - the workspace. Possible values: `reflink`, `symlink`, `hardlink`, `copy` or an - ordered combination of those, separated by commas e.g: - `reflink,hardlink,copy`. Default: `reflink,copy` - - - - There are pros and cons to different link types. Refer to [File link types] - for a full explanation of each one. - - - - If you set `cache.type` to `hardlink` or `symlink`, manually modifying tracked - data files in the workspace would corrupt the cache. To prevent this, DVC - automatically protects those kinds of links (making them read-only). Use - `dvc unprotect` to be able to modify them safely. - - To apply changes to this config option in the workspace, restore all file - links/copies from cache with `dvc checkout --relink`. - -- `cache.slow_link_warning` - used to turn off the warnings about having a slow - cache link type. These warnings are thrown by `dvc pull` and `dvc checkout` - when linking files takes longer than usual, to remind them that there are - faster cache link types available than the defaults (`reflink,copy` – see - `cache.type`). Accepts values `true` and `false`. - - - - These warnings are automatically turned off when `cache.type` is manually set. - - - -- `cache.shared` - permissions for newly created or downloaded cache files and - directories. The only accepted value right now is `group`, which makes DVC use - `664` (rw-rw-r--) for files and `775` (rwxrwxr-x) for directories. This is - useful when [sharing a cache] among projects. The default permissions for - cache files is system dependent. In Linux and macOS for example, they're - determined using [`os.umask`]. - -[file link types]: - /doc/user-guide/large-dataset-optimization#file-link-types-for-the-dvc-cache -[sharing a cache]: /doc/user-guide/how-to/share-a-dvc-cache -[`os.umask`]: https://docs.python.org/3/library/os.html#os.umask - -The following parameters allow setting an [external cache] location. A -`dvc remote` name is used (instead of the URL) because often it's necessary to -configure authentication or other connection settings, and configuring a remote -is the way that can be done. - -[external cache]: - /doc/user-guide/data-management/managing-external-data#setting-up-an-external-cache - -- `cache.local` - name of a [local remote] to use as external cache. This will - overwrite the value in `cache.dir` (see `dvc cache dir`). - -- `cache.s3` - name of an Amazon S3 remote to use as external cache. - -- `cache.gs` - name of a Google Cloud Storage remote to use as external cache. - -- `cache.ssh` - name of an SSH remote to use as external cache. - -- `cache.hdfs` - name of an HDFS remote to use as external cache. - -- `cache.webhdfs` - name of an HDFS remote with WebHDFS enabled to use as - external cache. - - - - Avoid using the same [remote storage] used for `dvc push` and `dvc pull` as - external cache, because it may cause file hash overlaps: the hash of an - external output could collide with that of a local file with - different content. - - [remote storage]: /doc/user-guide/data-management/remote-storage - - - -[local remote]: - /doc/user-guide/data-management/remote-storage#file-systems-local-remotes - -### exp - -Sets the default paths assumed by `dvc exp init`. This can help avoid overriding -them repeatedly with that command's options, for example if all of your -experiments or projects use a similar structure. - -- `exp.code` - path to your source file or directory dependency. - -- `exp.params` - path to your parameters file. - -- `exp.data` - path to your data file or directory dependency. - -- `exp.models` - path to your model/artifact(s) file or directory - output. - -- `exp.metrics` - path to your metrics file output. - -- `exp.plots` - path to your plots file or directory output. - -- `exp.live` - path to your [DVCLive](/doc/dvclive) output logs. - -### hydra - -Sets the defaults for experiment configuration via [Hydra -Composition]. - -- `hydra.enabled` - enables Hydra [config composition]. -- `hydra.config_dir` - location of the directory containing Hydra [config - groups]. Defaults to `conf`. -- `hydra.config_name` - the name of the file containing the Hydra [defaults - list] (located inside `hydra.config_dir`). Defaults to `config.yaml`. - -[config composition]: - https://hydra.cc/docs/tutorials/basic/your_first_app/composition/ -[config groups]: - https://hydra.cc/docs/tutorials/basic/your_first_app/config_groups/ -[defaults list]: https://hydra.cc/docs/tutorials/basic/your_first_app/defaults/ - -### parsing - -- `parsing.bool` - Controls the templating syntax for boolean values when used - in [dictionary unpacking]. - - Valid values are `"store_true"` (default) and `"boolean_optional"`, named - after [Python `argparse` actions]. - - Given the following `params.yaml`: - - ```yaml - dict: - bool-true: true - bool-false: false - ``` - - And corresponding `dvc.yaml`: - - ```yaml - stages: - foo: - cmd: python foo.py ${dict} - ``` - - When using `store_true`, `cmd` will be: - - ```shell - python foo.py --bool-true - ``` - - Whereas when using `boolean_optional`, `cmd` will be: - - ```shell - python foo.py --bool-true --no-bool-false - ``` - -- `parsing.list` - Controls the templating syntax for list values when used in - [dictionary unpacking]. - - Valid values are `"nargs"` (default) and `"append"`, named after [Python - `argparse` actions]. - - Given the following `params.yaml`: - - ```yaml - dict: - list: [1, 2, 'foo'] - ``` - - And corresponding `dvc.yaml`: - - ```yaml - stages: - foo: - cmd: python foo.py ${dict} - ``` - - When using `nargs`, `cmd` will be: - - ```shell - python foo.py --list 1 2 'foo' - ``` - - Whereas when using `append`, `cmd` will be: - - ```shell - python foo.py --list 1 --list 2 --list 'foo' - ``` - -[python `argparse` actions]: - https://docs.python.org/3/library/argparse.html#action - -### plots - -- `plots.auto_open` - if `true`, DVC will automatically open the HTML file - generated by `dvc plots` commands in a browser. `false` by default - -- `plots.html_template` - sets a - [custom HTML template](/doc/command-reference/plots/show#custom-html-templates) - for `dvc plots`. Accepts a path relative to the `.dvc/` folder. - -- `plots.out_dir` - changes the default value for `dvc plots show --out` and - `dvc plots diff --out`. The original default value is `dvc_plots`. - -### state - -- `state.row_limit` - maximum number of entries in state databases. This affects - the physical size of the state files, as well as the performance of certain - DVC operations. The default is 10,000,000 rows. The bigger the limit, the - longer the file hash history that DVC can keep, for example. - -- `state.row_cleanup_quota` - percentage of the state database to be deleted - when it reaches the `state.row_limit`. The default quota is 50%. DVC removes - the oldest entries (created when `dvc status` is used, for example). - -- `state.dir` - specify a custom location for the state databases (`links/` and - `md5/` directories), by default in `.dvc/tmp`. This may be necessary when - using DVC on NFS or other mounted volumes where SQLite encounters file - permission errors. - -### index - -- `index.dir` - specify a custom location for the directory where remote index - files will be stored, by default in `.dvc/tmp/index`. This may be necessary - when using DVC on NFS or other mounted volumes. - ## Example: Add an S3 remote, and set it as default > 💡 Before adding an S3 remote, be sure to diff --git a/content/docs/command-reference/exp/init.md b/content/docs/command-reference/exp/init.md index 8aa848b833..02407effa1 100644 --- a/content/docs/command-reference/exp/init.md +++ b/content/docs/command-reference/exp/init.md @@ -41,7 +41,8 @@ Creating train stage in dvc.yaml `dvc exp init` also generates the boilerplate project structure, including input files/directories and directories needed for future outputs. These locations can also be customized via [CLI options](#options) or interactive mode, or with -[configuration](/doc/command-reference/config#exp). Default structure: +[configuration](/doc/user-guide/project-structure/configuration#exp). Default +structure: ``` ├── data/ diff --git a/content/docs/command-reference/init.md b/content/docs/command-reference/init.md index 3868d2bb24..0ba868dbc8 100644 --- a/content/docs/command-reference/init.md +++ b/content/docs/command-reference/init.md @@ -132,10 +132,12 @@ automatic creation and updating of `.gitignore` files on `dvc add` or `dvc stage add`, as well as `dvc diff` and `dvc metrics diff`, which require Git revisions to compare. -DVC sets the `core.no_scm` config option value to `true` in the DVC -[config](/doc/command-reference/config) when initialized this way. This means -that even if the project is tracked by Git, or if Git is initialized in it -later, DVC will keep operating detached from Git in this project. +DVC sets the `core.no_scm` config option value to `true` in the [DVC +configuration] when initialized this way. This means that even if the project is +tracked by Git, or if Git is initialized in it later, DVC will keep operating +detached from Git in this project. + +[dvc configuration]: /doc/user-guide/project-structure/configuration ## Options diff --git a/content/docs/command-reference/plots/diff.md b/content/docs/command-reference/plots/diff.md index 3b662104f4..5c99f3750c 100644 --- a/content/docs/command-reference/plots/diff.md +++ b/content/docs/command-reference/plots/diff.md @@ -67,7 +67,8 @@ all the current plots, without comparisons. - `-o , --out ` - specify a directory to write the HTML file containing the plots. The default is `dvc_plots` or the value set with the - [`plots.out_dir`](/doc/command-reference/config#plots) config option. + [`plots.out_dir`](/doc/user-guide/project-structure/configuration#plots) + config option. - `-t , --template ` - [plot template](/doc/user-guide/experiment-management/visualizing-plots#plot-templates-data-series-only) diff --git a/content/docs/command-reference/plots/show.md b/content/docs/command-reference/plots/show.md index c69cb52f2c..4b532328c9 100644 --- a/content/docs/command-reference/plots/show.md +++ b/content/docs/command-reference/plots/show.md @@ -51,7 +51,8 @@ The default behavior of this command can be modified per [stage plot] file with - `-o , --out ` - specify a directory to write the HTML file containing the plots. The default is `dvc_plots` or the value set with the - [`plots.out_dir`](/doc/command-reference/config#plots) config option. + [`plots.out_dir`](/doc/user-guide/project-structure/configuration#plots) + config option. - `-t , --template ` - [plot template](/doc/user-guide/experiment-management/visualizing-plots#plot-templates-data-series-only) @@ -471,8 +472,8 @@ JavaScript libraries we downloaded above. We can use it like this: $ dvc plots show --html-template .dvc/plots/mypage.html ``` -You can also make it the default HTML template by setting it as `dvc config` -parameter `plots.html_template`. +You can also make it the default HTML template by setting it with DVC config +option [`plots.html_template`]. ```cli $ dvc config plots.html_template plots/mypage.html @@ -481,6 +482,8 @@ $ dvc config plots.html_template plots/mypage.html Note that the path supplied to `dvc config plots.html_template` is relative to `.dvc/` directory. +[`plots.html_template`]: /doc/user-guide/project-structure/configuration#plots + ## Example: Smooth plot In some cases we would like to smooth our plot. In this example we will use a diff --git a/content/docs/command-reference/remote/add.md b/content/docs/command-reference/remote/add.md index 7c14204b38..34bfdf1edf 100644 --- a/content/docs/command-reference/remote/add.md +++ b/content/docs/command-reference/remote/add.md @@ -52,7 +52,7 @@ The storage type determines which config parameters you can access via -This command creates a [`remote`] section in the project's [config file] +This command creates a [`remote`] section in the project's config file (`.dvc/config`). The `--default` (`-d`) flag uses the [`core`] config section: ```cli @@ -67,9 +67,8 @@ $ dvc remote add -d temp /tmp/dvcstore remote = myremote ``` -[`remote`]: /doc/command-reference/config#remote -[config file]: /doc/command-reference/config -[`core`]: /doc/command-reference/config#core +[`remote`]: /doc/user-guide/project-structure/configuration#remote +[`core`]: /doc/user-guide/project-structure/configuration#core diff --git a/content/docs/command-reference/remote/default.md b/content/docs/command-reference/remote/default.md index b05daa1fd4..b7688cee41 100644 --- a/content/docs/command-reference/remote/default.md +++ b/content/docs/command-reference/remote/default.md @@ -24,10 +24,10 @@ specified, this command returns the name of the default remote. $ dvc remote default myremote ``` -This command assigns the default remote in the `core` section of the DVC [config -file]. +This command assigns the default remote in the [`core`] section of the DVC +config file. -[config file]: /doc/command-reference/config +[`core`]: doc/user-guide/project-structure/configuration#core ```ini [core] @@ -38,8 +38,8 @@ Default remotes are expected by commands that accept a `-r`/`--remote` option (`dvc pull`, `dvc push`, `dvc status`, `dvc gc`, `dvc fetch`) when that option is omitted. -You can also use `dvc config`, `dvc remote add` and `dvc remote modify` commands -to set/unset/change the default remote configurations. +You can also use `dvc config`, `dvc remote add`, or `dvc remote modify` commands +to set/unset/change the default remote. Remotes are read from the system, global, project, and local config files (in that order). diff --git a/content/docs/command-reference/remote/index.md b/content/docs/command-reference/remote/index.md index 58ca1a1d9e..7ddfdd9280 100644 --- a/content/docs/command-reference/remote/index.md +++ b/content/docs/command-reference/remote/index.md @@ -46,7 +46,7 @@ default). Alternatively, the config files can be edited manually. [types of storage]: /doc/user-guide/data-management/remote-storage#supported-storage-types -[config files]: /doc/command-reference/config +[config files]: /doc/user-guide/project-structure/configuration ## Options diff --git a/content/docs/command-reference/remote/list.md b/content/docs/command-reference/remote/list.md index 3c282f4f3b..408bf64b9c 100644 --- a/content/docs/command-reference/remote/list.md +++ b/content/docs/command-reference/remote/list.md @@ -15,7 +15,7 @@ Reads [DVC configuration] and prints the list of available remotes, including their names and URLs/paths. Remotes are read from the system, global, project, and local config files (in that order). -[dvc configuration]: /doc/command-reference/config#remote +[dvc configuration]: /doc/user-guide/project-structure/configuration#remote ## Options diff --git a/content/docs/command-reference/remote/modify.md b/content/docs/command-reference/remote/modify.md index e3b3fe97d3..fb99531e1b 100644 --- a/content/docs/command-reference/remote/modify.md +++ b/content/docs/command-reference/remote/modify.md @@ -59,8 +59,8 @@ $ pip install "dvc[s3]" -[`remote`]: /doc/command-reference/config#remote -[config file]: /doc/command-reference/config +[`remote`]: /doc/user-guide/project-structure/configuration#remote +[config file]: /doc/user-guide/project-structure/configuration ## Command options/flags diff --git a/content/docs/command-reference/remote/remove.md b/content/docs/command-reference/remote/remove.md index 500a892ef7..3d38fecf9f 100644 --- a/content/docs/command-reference/remote/remove.md +++ b/content/docs/command-reference/remote/remove.md @@ -4,8 +4,8 @@ Remove a `dvc remote`. -This command affects DVC configuration files only. It does not physically remove -data files stored remotely. See `dvc gc --cloud` for that. +This command affects [DVC configuration] files only. It does not physically +remove data files stored remotely. See `dvc gc --cloud` for that. @@ -23,12 +23,13 @@ positional arguments: ## Description -This command removes a section in the DVC -[config file](/doc/command-reference/config). Alternatively, it is possible to -edit config files manually. +This command removes a section in the [DVC configuration] file. Alternatively, +it is possible to edit config files manually. The `name` argument is required. +[dvc configuration]: /doc/user-guide/project-structure/configuration + ## Options - `--system` - modify the system config file (e.g. `/etc/xdg/dvc/config`) diff --git a/content/docs/command-reference/remote/rename.md b/content/docs/command-reference/remote/rename.md index 7aee86050e..a64a82c935 100644 --- a/content/docs/command-reference/remote/rename.md +++ b/content/docs/command-reference/remote/rename.md @@ -23,9 +23,8 @@ positional arguments: ## Description -This command modify a section in the DVC -[config file](/doc/command-reference/config). Alternatively, it is possible to -edit config files manually. +This command modify a section in the [DVC configuration] file. Alternatively, it +is possible to edit config files manually. See also `dvc remote modify` to change other aspects of remote configuration, such as the URL or access credentials. @@ -33,6 +32,8 @@ such as the URL or access credentials. Both `name` and `new` arguments are required, with the old and new names for the DVC remote, respectively. +[dvc configuration]: /doc/user-guide/project-structure/configuration + ## Options - `--system` - modify the system config file (e.g. `/etc/xdg/dvc/config`) diff --git a/content/docs/sidebar.json b/content/docs/sidebar.json index d668fca062..0da64a4add 100644 --- a/content/docs/sidebar.json +++ b/content/docs/sidebar.json @@ -113,6 +113,7 @@ "label": ".dvcignore Files", "slug": "dvcignore-files" }, + "configuration", { "label": "Internal Files", "slug": "internal-files" diff --git a/content/docs/user-guide/analytics.md b/content/docs/user-guide/analytics.md index ac74893cf1..2c5f1ec686 100644 --- a/content/docs/user-guide/analytics.md +++ b/content/docs/user-guide/analytics.md @@ -54,7 +54,7 @@ DVC's analytics are sent through DVC's proxy to Google Analytics over HTTPS. DVC analytics help the entire community, so leaving it on is appreciated. However, if you want to opt out of DVC's analytics, you can disable it via -`dvc config` command: +`dvc config` command (or setting [`core.analytics`] manually): ```cli $ dvc config core.analytics false @@ -66,3 +66,5 @@ the active user or for everyone in the system, respectively. Another way to disable analytics is to set the environment variable `DVC_NO_ANALYTICS` (will override config). + +[`core.analytics`]: /doc/user-guide/project-structure/configuration#core diff --git a/content/docs/user-guide/data-management/remote-storage/index.md b/content/docs/user-guide/data-management/remote-storage/index.md index 67f5f01129..990eb2d2df 100644 --- a/content/docs/user-guide/data-management/remote-storage/index.md +++ b/content/docs/user-guide/data-management/remote-storage/index.md @@ -32,7 +32,7 @@ read and write to the [`remote`] section of the project's config file For example, let's define a remote storage location on an S3 bucket: -[`remote`]: /doc/command-reference/config#remote +[`remote`]: /doc/user-guide/project-structure/configuration#remote ```cli $ dvc remote add myremote s3://mybucket diff --git a/content/docs/user-guide/experiment-management/hydra-composition.md b/content/docs/user-guide/experiment-management/hydra-composition.md index 38923a31ac..09eceeb08b 100644 --- a/content/docs/user-guide/experiment-management/hydra-composition.md +++ b/content/docs/user-guide/experiment-management/hydra-composition.md @@ -74,8 +74,8 @@ defaults: -Use [`dvc config hydra`](/doc/command-reference/config#hydra) options to change -the default locations for the config groups directory and defaults list file. +Use `dvc config hydra` options to change the default locations for the config +groups directory and defaults list file. diff --git a/content/docs/user-guide/how-to/share-a-dvc-cache.md b/content/docs/user-guide/how-to/share-a-dvc-cache.md index c41aaab879..2111eb5338 100644 --- a/content/docs/user-guide/how-to/share-a-dvc-cache.md +++ b/content/docs/user-guide/how-to/share-a-dvc-cache.md @@ -7,10 +7,10 @@ description: >- # How to Share a DVC Cache -[Configuring](/doc/command-reference/config#cache) the project's -cache in a location shared by other projects helps avoid file -duplication and can reduce data transfers between different data locations. Some -scenarios: +[Configuring](/doc/user-guide/project-structure/configuration#cache) the +project's cache in a location shared by other projects helps avoid +file duplication and can reduce data transfers between different data locations. +Some scenarios: - Multiple users have copies of the same project on a single machine. - Different projects use the same data, so it makes sense to centralize its diff --git a/content/docs/user-guide/project-structure/configuration.md b/content/docs/user-guide/project-structure/configuration.md new file mode 100644 index 0000000000..22e1bf01ee --- /dev/null +++ b/content/docs/user-guide/project-structure/configuration.md @@ -0,0 +1,414 @@ +# DVC Configuration + +Once initialized in a project, DVC populates its installation +directory with [internal files], which include `.dvc/config`, the default +configuration file. + +Config files can be composed manually (or programmatically), or managed with the +helper command `dvc config`. + +[internal files]: /doc/user-guide/project-structure/internal-files + +## Config file locations + + + +`.dvc/config` is meant to be tracked by Git and should not contain sensitive +user info or secrets (passwords, SHH keys, etc). + + + +DVC supports saving configuration outside of the repository, either +in a Git-ignored file alongside the regular config file or in other places in +your file system. These locations and their loading priority are detailed below: + + + + +| Priority | Type | macOS location | Linux location (typical\*) | Windows location | +| -------- | ----------------- | ----------------------------------------------- | -------------------------- | --------------------------------------------------------- | +| 1 | Local | `.dvc/config.local` | _Same_ | _Same_ | +| 2 | Project (default) | `.dvc/config` | _Same_ | _Same_ | +| 3 | Global | `$HOME/Library/Application\ Support/dvc/config` | `$HOME/.config/dvc/config` | `%LocalAppData%\iterative\dvc\config` | +| 4 | System | `/Library/Application\ Support/dvc/config` | `/etc/xdg/dvc/config` | `%AllUsersProfile%\Application Data\iterative\dvc\config` | + + + +\* For Linux, the global file may be found in `$XDG_CONFIG_HOME`, and the system +file in `$XDG_CONFIG_DIRS[0]`, if those env vars are defined. + + + + + +See also `dvc config` flags `--local`, `--global`, and `--system`. + + + +## Configuration sections + +The following config sections are written by this command to the appropriate +config file (`.dvc/config` by default), supporting different config options +within: + +- [`core`](#core) - main section with the general config options +- [`remote`](#remote) - sections in the config file that describe [remote + storage] +- [`cache`](#cache) - options that affect the project's cache +- [`exp`](#exp) - options to change the default repo paths assumed by + `dvc exp init` +- [`hydra`](#hydra) - options around [Hydra Composition] for experiment + configuration. +- [`parsing`](#parsing) - options around the parsing of [dictionary unpacking]. +- [`plots`](#plots) - options for configuring `dvc plots`. +- [`state`](#state) - see [Internal directories and files][internals] to learn + more about the state database. +- [`index`](#index) - see [Internal directories and files][internals] to learn + more about remote index files. + +[remote storage]: /doc/user-guide/data-management/remote-storage +[hydra composition]: /doc/user-guide/experiment-management/hydra-composition +[dictionary unpacking]: + /doc/user-guide/project-structure/dvcyaml-files#dictionary-unpacking +[internals]: /doc/user-guide/project-structure/internal-files + +
+ +## core + +- [`core.remote`](#remote) - name of the default remote storage + +- `core.interactive` - whether to always ask for confirmation before reproducing + each [stage](/doc/command-reference/run) in `dvc repro`. (Normally, this + behavior requires using the `-i` option of that command.) Accepts values: + `true` and `false`. + +- `core.analytics` - used to turn off + [anonymized usage statistics](/doc/user-guide/analytics). Accepts values + `true` (default) and `false`. + +- `core.checksum_jobs` - number of threads for computing file hashes. Accepts + positive integers. The default value is `max(1, min(4, cpu_count() // 2))`. + +- `core.hardlink_lock` - use hardlink file locks instead of the default ones, + based on [`flock`](https://linux.die.net/man/2/flock) (i.e. project lock file + `.dvc/lock`). Accepts values `true` and `false` (default). Useful when the DVC + project is on a file system that doesn't properly support file locking (e.g. + [NFS v3 and older](http://nfs.sourceforge.net/)). + +- `core.no_scm` - tells DVC to not expect or integrate with Git (even if the + project is initialized inside a Git repo). Accepts values `true` + and `false` (default). Set with the `--no-scm` option of `dvc init` + ([more details](/doc/command-reference/init#initializing-dvc-without-git)). + +- `core.check_update` - disable/enable DVC's automatic update checks, which + notify the user when a new version is available. Accepts values `true` + (default) and `false`. + +- `core.autostage` - if enabled, DVC will automatically stage (`git add`) + DVC files created or modified by DVC commands. The files will not + be committed. Accepts values `true` and `false` (default). + +
+ +
+ +## remote + +Unlike most other sections, configuration files may have more than one +`'remote'`. All of them require a unique `"name"` and a `url` value. They can +also specify `jobs`, `verify`, and many platform-specific key/value pairs like +`port` and `password`. + + + +See [Remote Storage Configuration] for more details. + +[remote storage configuration]: + /doc/user-guide/data-management/remote-storage#configuration + + + +For example, the following config file defines a `temp` remote in the local file +system (located in `/tmp/dvcstore`), and marked as default (via [`core`](#core) +section): + +```ini +['remote "temp"'] + url = /tmp/dvcstore +[core] + remote = temp +``` + +
+ +
+ +## cache + +- `cache.dir` - set/unset cache directory location. A correct value is either an + absolute path, or a path **relative to the config file location**. The default + value is `cache`, that resolves to `.dvc/cache` (relative to the project + config file location). + + + + See also the helper command `dvc cache dir` to intuitively set this config + option, properly transforming paths relative to the current working directory + into paths relative to the config file location. + + + +- `cache.type` - link type that DVC should use to link data files from cache to + the workspace. Possible values: `reflink`, `symlink`, `hardlink`, `copy` or an + ordered combination of those, separated by commas e.g: + `reflink,hardlink,copy`. Default: `reflink,copy` + + + + There are pros and cons to different link types. Refer to [File link types] + for a full explanation of each one. + + + + If you set `cache.type` to `hardlink` or `symlink`, manually modifying tracked + data files in the workspace would corrupt the cache. To prevent this, DVC + automatically protects those kinds of links (making them read-only). Use + `dvc unprotect` to be able to modify them safely. + + To apply changes to this config option in the workspace, restore all file + links/copies from cache with `dvc checkout --relink`. + +- `cache.slow_link_warning` - used to turn off the warnings about having a slow + cache link type. These warnings are thrown by `dvc pull` and `dvc checkout` + when linking files takes longer than usual, to remind them that there are + faster cache link types available than the defaults (`reflink,copy` – see + `cache.type`). Accepts values `true` and `false`. + + + + These warnings are automatically turned off when `cache.type` is manually set. + + + +- `cache.shared` - permissions for newly created or downloaded cache files and + directories. The only accepted value right now is `group`, which makes DVC use + `664` (rw-rw-r--) for files and `775` (rwxrwxr-x) for directories. This is + useful when [sharing a cache] among projects. The default permissions for + cache files is system dependent. In Linux and macOS for example, they're + determined using [`os.umask`]. + +[file link types]: + /doc/user-guide/large-dataset-optimization#file-link-types-for-the-dvc-cache +[sharing a cache]: /doc/user-guide/how-to/share-a-dvc-cache +[`os.umask`]: https://docs.python.org/3/library/os.html#os.umask + +The following parameters allow setting an [external cache] location. A +`dvc remote` name is used (instead of the URL) because often it's necessary to +configure authentication or other connection settings, and configuring a remote +is the way that can be done. + +[external cache]: + /doc/user-guide/data-management/managing-external-data#setting-up-an-external-cache + +- `cache.local` - name of a [local remote] to use as external cache. This will + overwrite the value in `cache.dir` (see `dvc cache dir`). + +- `cache.s3` - name of an Amazon S3 remote to use as external cache. + +- `cache.gs` - name of a Google Cloud Storage remote to use as external cache. + +- `cache.ssh` - name of an SSH remote to use as external cache. + +- `cache.hdfs` - name of an HDFS remote to use as external cache. + +- `cache.webhdfs` - name of an HDFS remote with WebHDFS enabled to use as + external cache. + + + + Avoid using the same [remote storage] used for `dvc push` and `dvc pull` as + external cache, because it may cause file hash overlaps: the hash of an + external output could collide with that of a local file with + different content. + + [remote storage]: /doc/user-guide/data-management/remote-storage + + + +[local remote]: + /doc/user-guide/data-management/remote-storage#file-systems-local-remotes + +
+ +
+ +## exp + +Sets the default paths assumed by `dvc exp init`. This can help avoid overriding +them repeatedly with that command's options, for example if all of your +experiments or projects use a similar structure. + +- `exp.code` - path to your source file or directory dependency. + +- `exp.params` - path to your parameters file. + +- `exp.data` - path to your data file or directory dependency. + +- `exp.models` - path to your model/artifact(s) file or directory + output. + +- `exp.metrics` - path to your metrics file output. + +- `exp.plots` - path to your plots file or directory output. + +- `exp.live` - path to your [DVCLive](/doc/dvclive) output logs. + +
+ +
+ +### hydra + +Sets the defaults for experiment configuration via [Hydra +Composition]. + +- `hydra.enabled` - enables Hydra [config composition]. +- `hydra.config_dir` - location of the directory containing Hydra [config + groups]. Defaults to `conf`. +- `hydra.config_name` - the name of the file containing the Hydra [defaults + list] (located inside `hydra.config_dir`). Defaults to `config.yaml`. + +[config composition]: + https://hydra.cc/docs/tutorials/basic/your_first_app/composition/ +[config groups]: + https://hydra.cc/docs/tutorials/basic/your_first_app/config_groups/ +[defaults list]: https://hydra.cc/docs/tutorials/basic/your_first_app/defaults/ + +
+ +
+ +## parsing + +- `parsing.bool` - Controls the templating syntax for boolean values when used + in [dictionary unpacking]. + + Valid values are `"store_true"` (default) and `"boolean_optional"`, named + after [Python `argparse` actions]. + + Given the following `params.yaml`: + + ```yaml + dict: + bool-true: true + bool-false: false + ``` + + And corresponding `dvc.yaml`: + + ```yaml + stages: + foo: + cmd: python foo.py ${dict} + ``` + + When using `store_true`, `cmd` will be: + + ```shell + python foo.py --bool-true + ``` + + Whereas when using `boolean_optional`, `cmd` will be: + + ```shell + python foo.py --bool-true --no-bool-false + ``` + +- `parsing.list` - Controls the templating syntax for list values when used in + [dictionary unpacking]. + + Valid values are `"nargs"` (default) and `"append"`, named after [Python + `argparse` actions]. + + Given the following `params.yaml`: + + ```yaml + dict: + list: [1, 2, 'foo'] + ``` + + And corresponding `dvc.yaml`: + + ```yaml + stages: + foo: + cmd: python foo.py ${dict} + ``` + + When using `nargs`, `cmd` will be: + + ```shell + python foo.py --list 1 2 'foo' + ``` + + Whereas when using `append`, `cmd` will be: + + ```shell + python foo.py --list 1 --list 2 --list 'foo' + ``` + +[python `argparse` actions]: + https://docs.python.org/3/library/argparse.html#action + +
+ +
+ +## plots + +- `plots.auto_open` - if `true`, DVC will automatically open the HTML file + generated by `dvc plots` commands in a browser. `false` by default + +- `plots.html_template` - sets a + [custom HTML template](/doc/command-reference/plots/show#custom-html-templates) + for `dvc plots`. Accepts a path relative to the `.dvc/` folder. + +- `plots.out_dir` - changes the default value for `dvc plots show --out` and + `dvc plots diff --out`. The original default value is `dvc_plots`. + +
+ +
+ +## state + +- `state.row_limit` - maximum number of entries in state databases. This affects + the physical size of the state files, as well as the performance of certain + DVC operations. The default is 10,000,000 rows. The bigger the limit, the + longer the file hash history that DVC can keep, for example. + +- `state.row_cleanup_quota` - percentage of the state database to be deleted + when it reaches the `state.row_limit`. The default quota is 50%. DVC removes + the oldest entries (created when `dvc status` is used, for example). + +- `state.dir` - specify a custom location for the state databases (`links/` and + `md5/` directories), by default in `.dvc/tmp`. This may be necessary when + using DVC on NFS or other mounted volumes where SQLite encounters file + permission errors. + +
+ +
+ +## index + +- `index.dir` - specify a custom location for the directory where remote index + files will be stored, by default in `.dvc/tmp/index`. This may be necessary + when using DVC on NFS or other mounted volumes. + +
diff --git a/content/docs/user-guide/project-structure/index.md b/content/docs/user-guide/project-structure/index.md index c4506eb8fb..24a42f01f8 100644 --- a/content/docs/user-guide/project-structure/index.md +++ b/content/docs/user-guide/project-structure/index.md @@ -16,9 +16,11 @@ will create and manage different DVC metafiles (below), and populate the - `.dvcignore` files (optional) contain a list of paths for DVC to ignore, which can dramatically increase its operational performance. -- Internal files and directories in `.dvc/` contains the local - [configuration](/doc/command-reference/config) file(s), default local cache - location, and other utilities that DVC needs to operate. +- Internal files and directories in `.dvc/` contain the local [configuration] + file(s), default local cache location, and other utilities that DVC needs to + operate. + +[configuration]: /doc/user-guide/project-structure/configuration diff --git a/content/docs/user-guide/project-structure/internal-files.md b/content/docs/user-guide/project-structure/internal-files.md index 27ba045cc1..05d01a8314 100644 --- a/content/docs/user-guide/project-structure/internal-files.md +++ b/content/docs/user-guide/project-structure/internal-files.md @@ -4,16 +4,27 @@ Once initialized in a project, DVC populates its installation directory (`.dvc/`) with the internal directories and files needed for DVC operation. -⚠️ Not to be confused with `.dvc` files. + + +Not to be confused with `.dvc` files. + + -- `.dvc/config`: This is a configuration file. The config file can be edited by - hand or with the `dvc config` command. +- `.dvc/config`: This is the default [DVC configuration] file. It can be edited + by hand or with the `dvc config` command. -- `.dvc/config.local`: This is a local configuration file, that will overwrite - options in `.dvc/config`. This is useful when you need to specify private - options in your config that you don't want to track and share through Git - (credentials, private locations, etc). The local config file can be edited by - hand or with the command `dvc config --local`. +- `.dvc/config.local`: This is an optional Git-ignored configuration file, that + will overwrite options in `.dvc/config`. This is useful when you need to + specify sensitive values (secrets) which should not reach the Git repo + (credentials, private locations, etc). This config file can also be edited by + hand or with `dvc config --local`. + + + + See more + [config file locations](/doc/user-guide/project-structure/configuration#config-file-locations). + + - `.dvc/cache`: Default location of the cache directory. The cache stores the project data in a special @@ -24,10 +35,14 @@ operation. See `dvc config cache` for related configuration options, including changing its location. - > Note that DVC includes the cache directory in `.gitignore` during - > initialization. No data tracked by DVC should ever be pushed to the Git - > repository, only the DVC files that are needed to download or - > reproduce that data. + + + Note that DVC includes the cache directory in `.gitignore` during + initialization. No data tracked by DVC should ever be pushed to the Git + repository, only the DVC files that are needed to download or + reproduce that data. + + - `.dvc/cache/runs`: Default location of the [run cache](#run-cache). @@ -39,20 +54,32 @@ operation. - `.dvc/tmp/index`: Directory for remote index files that are used for optimizing `dvc push`, `dvc pull`, `dvc fetch` and `dvc status -c` operations. - > This location may be overridden with `dvc config index.dir`. + + + This location may be overridden with `dvc config index.dir`. + + - `.dvc/tmp/md5s`: This directory is used for optimization. It contains a SQLite state database that stores hash values for files tracked in a DVC project. It also saves the corresponding timestamps and inodes to avoid unnecessary file hash computations. - > This parent location may be overridden with `dvc config state.dir`. + + + This parent location may be overridden with `dvc config state.dir`. + + - `.dvc/tmp/links`: This directory is used to clean up your workspace when calling `dvc checkout`. It contains a SQLite state database that stores a list of file links created by DVC (from cache to workspace). - > This parent location may be overridden with `dvc config state.dir`. + + + This parent location may be overridden with `dvc config state.dir`. + + - `.dvc/tmp/updater`: This file is used to store the latest available version of DVC. It's used to remind the user to upgrade when the installed version is @@ -69,6 +96,7 @@ operation. - `.dvc/tmp/exps`: This directory will contain workspace copies used for temporary or [queued experiments]. +[dvc configuration]: /doc/user-guide/project-structure/configuration [queued experiments]: /doc/user-guide/experiment-management/running-experiments#the-experiments-queue diff --git a/content/docs/user-guide/troubleshooting.md b/content/docs/user-guide/troubleshooting.md index 9639e3bd8b..9b69ab5156 100644 --- a/content/docs/user-guide/troubleshooting.md +++ b/content/docs/user-guide/troubleshooting.md @@ -51,9 +51,9 @@ in the correct `region` and/or `endpointurl` (see `dvc remote modify`). ## Unable to detect cache type {#no-dvc-cache} Unable to detect supported link types, as the -[cache directory](/doc/command-reference/config#cache) doesn't exist. It is -usually created automatically by DVC commands that need it, but you can create -it manually (e.g. `mkdir .dvc/cache`) to enable this check. +[cache directory](/doc/user-guide/project-structure/configuration#cache) doesn't +exist. It is usually created automatically by DVC commands that need it, but you +can create it manually (e.g. `mkdir .dvc/cache`) to enable this check. ## Unable to acquire lock {#lock-issue}