guide: Data Management

content/docs/command-reference/config.md

@@ -165,24 +165,31 @@ See `dvc remote add` and `dvc remote modify` for more information.
   value is `cache`, that resolves to `.dvc/cache` (relative to the project
   config file location).
 
-  > See also the helper command `dvc cache dir` to intuitively set this config
-  > option, properly transforming paths relative to the current working
-  > directory into paths relative to the config file location.
+  <admon type="tip">
+
+  See also the helper command `dvc cache dir` to intuitively set this config
+  option, properly transforming paths relative to the current working directory
+  into paths relative to the config file location.
+
+  </admon>
 
 - `cache.type` - link type that DVC should use to link data files from cache to
   the workspace. Possible values: `reflink`, `symlink`, `hardlink`, `copy` or an
   ordered combination of those, separated by commas e.g:
   `reflink,hardlink,copy`. Default: `reflink,copy`
 
+  <admon type="info">
+
+  There are pros and cons to different link types. Refer to [File link types]
+  for a full explanation of each one.
+
+  </admon>
+
   If you set `cache.type` to `hardlink` or `symlink`, manually modifying tracked
   data files in the workspace would corrupt the cache. To prevent this, DVC
   automatically protects those kinds of links (making them read-only). Use
   `dvc unprotect` to be able to modify them safely.
 
-  There are pros and cons to different link types. Refer to
-  [File link types](/doc/user-guide/data-management/large-dataset-optimization#file-link-types-for-the-dvc-cache)
-  for a full explanation of each one.
-
   To apply changes to this config option in the workspace, restore all file
   links/copies from cache with `dvc checkout --relink`.
 
@@ -192,16 +199,23 @@ See `dvc remote add` and `dvc remote modify` for more information.
   faster cache link types available than the defaults (`reflink,copy` – see
   `cache.type`). Accepts values `true` and `false`.
 
-  > These warnings are automatically turned off when `cache.type` is manually
-  > set.
+  <admon type="info">
+
+  These warnings are automatically turned off when `cache.type` is manually set.
+
+  </admon>
 
 - `cache.shared` - permissions for newly created or downloaded cache files and
   directories. The only accepted value right now is `group`, which makes DVC use
   `664` (rw-rw-r--) for files and `775` (rwxrwxr-x) for directories. This is
-  useful when [sharing a cache](/doc/user-guide/how-to/share-a-dvc-cache) among
-  projects. The default permissions for cache files is system dependent. In
-  Linux and macOS for example, they're determined using
-  [`os.umask`](https://docs.python.org/3/library/os.html#os.umask).
+  useful when [sharing a cache] among projects. The default permissions for
+  cache files is system dependent. In Linux and macOS for example, they're
+  determined using [`os.umask`].
+
+[file link types]:
+  /doc/user-guide/large-dataset-optimization#file-link-types-for-the-dvc-cache
+[sharing a cache]: /doc/user-guide/how-to/share-a-dvc-cache
+[`os.umask`]: https://docs.python.org/3/library/os.html#os.umask
 
 The following parameters allow setting an
 [external cache](/doc/user-guide/data-management/managing-external-data#setting-up-an-external-cache)

content/docs/sidebar.json

@@ -122,7 +122,7 @@
       },
       {
         "slug": "data-management",
-        "source": false,
+        "source": "data-management/index.md",
         "children": [
           "large-dataset-optimization",
           "importing-external-data",

content/docs/use-cases/versioning-data-and-models/index.md

@@ -36,7 +36,7 @@ learn how DVC looks and feels firsthand.
 
 As you use DVC, unique versions of your data files and directories are
 [cached](/doc/user-guide/project-structure/internal-files#structure-of-the-cache-directory)
-in a systematic way (preventing file duplication). The working datastore is
+in a systematic way (preventing file duplication). The working data store is
 separated from your <abbr>workspace</abbr> to keep the project light, but stays
 connected via file
 [links](/doc/user-guide/data-management/large-dataset-optimization#file-link-types-for-the-dvc-cache)

content/docs/user-guide/basic-concepts/dvc-cache.md

@@ -3,7 +3,7 @@ name: 'DVC Cache'
 match: ['DVC cache', cache, caches, cached, 'cache directory', caching]
 tooltip: >-
   The DVC cache is a hidden storage (by default in `.dvc/cache`) for files and
-  directories tracked by DVC, and their different versions. Learn more about its
-  structure
-  [here](/doc/user-guide/project-structure/internal-files#structure-of-the-cache-directory).
+  directories tracked by DVC, and their different versions. For efficiency, it
+  uses a content-addressable
+  [structure](/doc/user-guide/project-structure/internal-files#structure-of-the-cache-directory).
 ---

content/docs/user-guide/data-management/index.md

@@ -0,0 +1,91 @@
+# Data Management with DVC
+
+DVC helps you manage and share arbitrarily large files, datasets, and ML models
+anywhere: mounted drives, network resources (e.g. NAS), external devices, or
+remotely (SSH servers, cloud storage, etc.). Once the project is configured, you
+can manipulate files normally in your local workspace. DVC tracks, restores, and
+synchronizes them across locations.
+
+![]() _Local, external, and remote storage locations_
+
+## The data cache
+
+<abbr>DVC projects</abbr> separate data from code by replacing large files, data
+artifacts, ML models, etc. in your <abbr>workspace</abbr> with small
+[metafiles]; We call this process _codification_ (of the data). The actual file
+contents are cached in an independent data store and linked to your project.
+
+![]() _Separating code from data_
+
+<admon type="info">
+
+In order to [avoid duplicate content], and to support
+[versioning features](#data-versioning), files and directories are reorganized
+in the cache into a [content-addressable structure].
+
+[avoid duplicate content]:
+  /doc/user-guide/data-management/large-dataset-optimization
+
+</admon>
+
+DVC expects fast storage access to the <abbr>cache</abbr>, so it's local to the
+project by default (found in `.dvc/cache`). It can, however, be moved to an
+external location in the file system or network, for example to [share it] among
+several projects.
+
+<admon type="tip">
+
+A DVC cache could even be set up in a remote system and accessed through the
+internet, but this is typically too slow for working with the data regularly.
+
+</admon>
+
+[metafiles]: /doc/user-guide/project-structure
+[share it]: /doc/user-guide/how-to/share-a-dvc-cache
+[content-addressable structure]:
+  /doc/user-guide/project-structure/internal-files#structure-of-the-cache-directory
+
+## Remote storage
+
+Optionally, DVC supports additional storage locations such as cloud services
+(Amazon S3, Google Drive, Azure Blob Storage, etc.), SSH servers, HDFS, and
+others. [DVC remotes] are typically used to sync copies of all or some of your
+datasets and models, for sharing or backup.
+
+![]() _Distributed collaboration on DVC projects_
+
+<admon type="info">
+
+Remote storage uses the same [structure][content-addressable structure] of the
+data cache.
+
+</admon>
+
+[dvc remotes]: /doc/command-reference/remote
+
+## Data Versioning
+
+DVC brings source code management (SCM) to data science. Specifically, the
+metafiles in your repo can be handled with standard [Git workflows] (commits,
+branching, pull requests, etc.). This way machine learning teams can apply
+mature software engineering practices.
+
+[git workflows]: https://www.atlassian.com/git/tutorials/comparing-workflows
+
+<admon icon="book">
+
+Refer to [Versioning Data and Models] to learn more.
+
+[versioning data and models]: /doc/use-cases/versioning-data-and-models
+
+</admon>
+
+<!--
+## Cloud versioning
+
+_New in DVC 2.30.0 (see `dvc version`)_
+
+To simplify remote data operations, DVC now supports native versioning of files
+and directories on several cloud providers. This means that you can browse your
+files normally as you would see them in your local workspace.
+-->

content/docs/user-guide/project-structure/internal-files.md

@@ -15,24 +15,26 @@ operation.
   (credentials, private locations, etc). The local config file can be edited by
   hand or with the command `dvc config --local`.
 
-- `.dvc/cache`: Default location of the <abbr>cache</abbr> directory. The cache
-  stores the project data in a special
-  [structure](#structure-of-the-cache-directory). The data files and directories
-  in the <abbr>workspace</abbr> will only contain links to the data files in the
-  cache (refer to
-  [Large Dataset Optimization](/doc/user-guide/data-management/large-dataset-optimization).
-  See `dvc config cache` for related configuration options, including changing
-  its location.
-
-  > Note that DVC includes the cache directory in `.gitignore` during
-  > initialization. No data tracked by DVC should ever be pushed to the Git
-  > repository, only the <abbr>DVC files</abbr> that are needed to download or
-  > reproduce that data.
+- `.dvc/cache`: Default location of the <abbr>cache directory</abbr>. The cache
+  stores the project data in a special content-addressable
+  [structure](#structure-of-the-cache-directory). The files and directories
+  visible in the <abbr>workspace</abbr> will typically be [links] to cached
+  data. See `dvc config cache` for related configuration options, including
+  changing this default location.
+
+  <admon type="info">
+
+  DVC includes the cache directory in `.gitignore` during [initialization]. No
+  data tracked by DVC should ever be pushed to the Git repository, only the
+  <abbr>DVC files</abbr> that are needed to download or reproduce that data.
+
+  [initialization]: /doc/command-reference/init
+
+  </admon>
 
 - `.dvc/cache/runs`: Default location of the [run-cache](#run-cache).
 
-- `.dvc/plots`: Directory for
-  [plot templates](/doc/user-guide/experiment-management/visualizing-plots#plot-templates-data-series-only)
+- `.dvc/plots`: Directory for [plot templates]
 
 - `.dvc/tmp`: Directory for miscellaneous temporary files
 
@@ -69,22 +71,34 @@ operation.
 - `.dvc/tmp/exps`: This directory will contain workspace copies used for
   temporary or [queued experiments].
 
+[links]: /doc/user-guide/large-dataset-optimization
+[plot templates]:
+  /doc/user-guide/visualizing-plots#plot-templates-data-series-only
 [queued experiments]:
   /doc/user-guide/experiment-management/running-experiments#the-experiments-queue
 
 ## Structure of the cache directory
 
-The DVC cache is a
-[content-addressable storage](https://en.wikipedia.org/wiki/Content-addressable_storage)
-(by default in `.dvc/cache`), which adds a layer of indirection between code and
-data.
+The DVC cache is a [content-addressable storage] (found in `.dvc/cache` by
+default), which adds a layer of [indirection] between code and data.
+
+[content-addressable storage]:
+  https://en.wikipedia.org/wiki/Content-addressable_storage
+[indirection]: https://en.wikipedia.org/wiki/Indirection
 
-There are two ways in which the data is <abbr>cached</abbr>, depending on
-whether it's a single file, or a directory (which may contain multiple files).
+<admon type="info">
 
-Note files are renamed, reorganized, and directory trees are flattened in the
-cache, which always has exactly one depth level with 2-character directories
-(based on hashes of the data contents, as explained next).
+This structure is also used by [remote storage].
+
+[remote storage]: /doc/user-guide/data-management#remote-storage
+
+</admon>
+
+There are two ways in which data is <abbr>cached</abbr>, depending on whether
+it's a single file or a folder: Files are renamed and reorganized to prevent
+duplication. Directory trees are flattened so that the cache always has exactly
+one depth level containing 2-character directories (based on hashes of the data
+contents, as explained next).
 
 ### Files
 
@@ -94,10 +108,14 @@ rest become the file name of the cached file. For example, if a data file has a
 hash value of `ec1d2935f811b77cc49b031b999cbf17`, its path in the cache will be
 `.dvc/cache/ec/1d2935f811b77cc49b031b999cbf17`.
 
-> Note that file hashes are calculated from file contents only. 2 or more files
-> with different names but the same contents can exist in the workspace and be
-> tracked by DVC, but only one copy is stored in the cache. This helps avoid
-> data duplication.
+<admon type="info">
+
+File hashes are calculated from file contents only. 2 or more files with
+different names but the same contents can exist in the workspace and be tracked
+by DVC, but only one copy is stored in the cache. This helps avoid data
+duplication.
+
+</admon>
 
 ### Directories