guide: remote storage #4058

jorgeorpinel · 2022-10-20T21:54:16Z

To finish addressing the 3rd check box in #2856 (comment). Begins addressing #2866. Planned structure:

What is remote storage
Types & configuration (not sure in which order)

Main file to review: UG/data-management/remote-storage-setup.md

In review app: https://dvc-org-guide-data-mgmt-4lzkrz.herokuapp.com/doc/user-guide/data-management/remote-storage

Closes ref: clarify about obfuscated remote file structure (same as cache) #1762, apparently.
Add more links after guide: Basic Operations (Data Mgmt) #4053 is merged.

Next PRs:

Remote set up by storage types ref: remote updates #4062

some updates around the topic in existing docs

per #4042 (review)

add prospective figure titles

github-actions · 2022-10-20T22:08:07Z

e5c6f13

Link Check Report

content/docs/user-guide/data-management/data-versioning.md
- doc/user-guide/how-to/share-a-dvc-cache = https://doc/user-guide/how-to/share-a-dvc-cache (EAI_AGAIN: request to https://doc/user-guide/how-to/share-a-dvc-cache failed, reason: getaddrinfo EAI_AGAIN doc)

1/22 links failed.

per #4042 (review)

and update fig. 1 per #4042 (review)

per #4042 (review)

moved from #4053 (review)

per #4042 (comment)

per #4042 (review)

github-actions · 2023-01-24T09:43:01Z

Link Check Report

There were no links to check!

Rel. #4042

jorgeorpinel · 2023-01-24T21:36:13Z

content/docs/sidebar.json

        "children": [
          "large-dataset-optimization",
+          "remote-storage",
+          "cloud-versioning",
          "importing-external-data",
-          "managing-external-data",
-          "cloud-versioning"
+          "managing-external-data"


Reordered this a little bit.

jorgeorpinel · 2023-01-24T21:40:07Z

Merging this since it can't hurt.

* guide: draft structure of Data Mgmt and some updates around the topic in existing docs * guide: full text for draft intro to DM * guide: hide cloud versioning info per #4042 (review) * guide: clarify Data Mgmt parts and add prospective figure titles * guide: add figure drafts to Data Mgmt * guide: SCM->VC (Data Mgmt) * guide: update 2 figs and add 1 more (Data Mgmt) * guide: roll back unrelated changes per #4042 (review) * guide: mention clouds first (DM) and and update fig. 1 per #4042 (review) * guide: flatten DM index per #4042 (review) * guide: udpates to DM/ DV moved from #4053 (review) * guide: add DM/ Data Versioning page per #4042 (comment) * guide: update outdated link * guide: revert more unrelatedly chaqnged files per #4042 (review) * guide: remove unused ref link * guide: DM/ Remote Storage (not just Setup) and and some links from cmd refs and avoid term "data remote" and some admons nearby... * guide: remove a comment * guide: draft for DM/ Remote Storage content * ref: expand config.remote and link to/from Remotes guide * ref: fix remote config file examples * guide: complete Remote Config section and and add Project config section to DM/ DV guide * ref: rewrite remote add and modify Descs * guide: complete list of supported storage types * ref: rewrite remote index page from extracted from #4053 * guide: clarify `remote modify` phrase in in the Remote config section of DM/ Remote Storage * Update content/docs/user-guide/data-management/data-versioning.md * guide: update versioning config per #4058 (review) * guide: don't call remote storage "additional" here (in the DM/ Remote Storage guide) per #4058 (review) Co-authored-by: Dave Berenbaum <[email protected]> * guide: pull -> download (DM/ RS intro) * guide: remove "optional" from Remote Storage nav & title per #4058 (review) * guide: splits and notes around Data Mgmt index page rel. #4042 (comment) * guide: Data Mgmt intro + note updates * guide: draft of all contents + + remove comments * guide: small impros to Data Mgmt in prep for #4042 (review) * guide: rewrite Data Mgmt index in before/after form per #4042 (review) * guide: add draft figure for Data Mgmt * guide: simplify/refocus data mgmt index per #4042 (review) * work around commented header bug * guide: drop DM/ DV page * guide: rewrite DM intro and - hide benefits (for now) - remove codification comment block * guide: use DM table instead of figure for now * guide: rewrite Data Mgmt story * guide: add draft figures to Data Mgmt * guide: simplify Data Mgmt story and benefits * guide: remove unused images (DM) * guide: update Data Mgmt figures (v1) * guide: rewrite text of Data Mgmt index * guide: update Data Mgmt figures * guide: iterate on Data Mgmt again * guide: update Data Mgmt figs * guide: more supporting info about Data Mgmt * guide: update figures (much more concrete) and and matching text updates * guide: edits to How it works (Data Mgmt) * guide: update Data Mgmt figures Rel. #4042 (comment) * guide: emphaisze dataset versions in UG fig 1 Rel. #4042 (comment) * guide: update Data Mgmt figures (with notes), expand img captions, and update text accordingly. * guide: more updates to text and figure styles, esp. to the first half and comment some stuff out (temporary) * guide: update figures and text (Data Mgmt) ... Using a tabs toggle for the 2nd fig. * guide: Data Management text (section 1) finalized for this version of figures * guide: Data Management (main text) finalized for this version of figures * guide: Data Management (secondary text) pending diagram and code sample(s) * guide: add DVC data mgmt technical diagram & dummy sample CLI blocks * guide: update Data Mgmt text * guide: udpate text and 2nd figure (Data Mgmt) * guide: draft 2nd and 3rd figures * guide: rewrite Data Mgmt/ How it works & and Benefits/ Tradeoffs Probably still unfinished... Missing more data versioning info? See HTML comments. * guide: update drafts of Data Mgmt figures 2, 3 * guide: Data Mgmt improvements and hide the benefits list for now * guide: separate from Data Mgmt work Rel. #4042 * Apply suggestions from code review * Merge branch main + * ref: bring cloud versioning copy edits of import-url from https://github.com/iterative/dvc.org/pull/4260/files#diff-ef95e18c4bd039757695065a23946dc27e28b4727ce07c670cdc096e34dbe3b3 * ref: clarify import-url with cloud versioning per #4142 (review) * ref: updates to import-url --version-aware and update --rev * ref: add import-url --version aware to Synopsis per #4089 (comment) * Restyled by prettier (#4266) Co-authored-by: Restyled.io <[email protected]> * ref: updates around worktree updates (cloud versioning) * ref: link from `remote` (index) to storage types * guide: roll back changes to dvc.yaml `rev` field spec * Update content/docs/command-reference/update.md * guide: link refs in .dvc file spec * Restyled by prettier (#4319) Co-authored-by: Restyled.io <[email protected]> * Update content/docs/command-reference/update.md --------- Co-authored-by: Dave Berenbaum <[email protected]> Co-authored-by: rogermparent <[email protected]> Co-authored-by: restyled-io[bot] <32688539+restyled-io[bot]@users.noreply.github.com> Co-authored-by: Restyled.io <[email protected]>

* guide: draft structure of Data Mgmt and some updates around the topic in existing docs * guide: full text for draft intro to DM * guide: hide cloud versioning info per #4042 (review) * guide: clarify Data Mgmt parts and add prospective figure titles * guide: add figure drafts to Data Mgmt * guide: SCM->VC (Data Mgmt) * guide: update 2 figs and add 1 more (Data Mgmt) * guide: roll back unrelated changes per #4042 (review) * guide: mention clouds first (DM) and and update fig. 1 per #4042 (review) * guide: flatten DM index per #4042 (review) * guide: udpates to DM/ DV moved from #4053 (review) * guide: add DM/ Data Versioning page per #4042 (comment) * guide: update outdated link * guide: revert more unrelatedly chaqnged files per #4042 (review) * guide: remove unused ref link * guide: DM/ Remote Storage (not just Setup) and and some links from cmd refs and avoid term "data remote" and some admons nearby... * guide: remove a comment * guide: draft for DM/ Remote Storage content * ref: expand config.remote and link to/from Remotes guide * ref: fix remote config file examples * guide: complete Remote Config section and and add Project config section to DM/ DV guide * ref: rewrite remote add and modify Descs * guide: complete list of supported storage types * ref: rewrite remote index page from extracted from #4053 * guide: clarify `remote modify` phrase in in the Remote config section of DM/ Remote Storage * Update content/docs/user-guide/data-management/data-versioning.md * guide: update versioning config per #4058 (review) * guide: don't call remote storage "additional" here (in the DM/ Remote Storage guide) per #4058 (review) Co-authored-by: Dave Berenbaum <[email protected]> * guide: pull -> download (DM/ RS intro) * guide: remove "optional" from Remote Storage nav & title per #4058 (review) * guide: splits and notes around Data Mgmt index page rel. #4042 (comment) * guide: Data Mgmt intro + note updates * guide: draft of all contents + + remove comments * guide: small impros to Data Mgmt in prep for #4042 (review) * guide: rewrite Data Mgmt index in before/after form per #4042 (review) * guide: add draft figure for Data Mgmt * guide: simplify/refocus data mgmt index per #4042 (review) * work around commented header bug * guide: drop DM/ DV page * guide: rewrite DM intro and - hide benefits (for now) - remove codification comment block * guide: use DM table instead of figure for now * guide: rewrite Data Mgmt story * guide: add draft figures to Data Mgmt * guide: simplify Data Mgmt story and benefits * guide: remove unused images (DM) * guide: update Data Mgmt figures (v1) * guide: rewrite text of Data Mgmt index * guide: update Data Mgmt figures * guide: iterate on Data Mgmt again * guide: update Data Mgmt figs * guide: more supporting info about Data Mgmt * guide: update figures (much more concrete) and and matching text updates * guide: edits to How it works (Data Mgmt) * guide: update Data Mgmt figures Rel. #4042 (comment) * guide: emphaisze dataset versions in UG fig 1 Rel. #4042 (comment) * guide: update Data Mgmt figures (with notes), expand img captions, and update text accordingly. * guide: more updates to text and figure styles, esp. to the first half and comment some stuff out (temporary) * guide: update figures and text (Data Mgmt) ... Using a tabs toggle for the 2nd fig. * guide: Data Management text (section 1) finalized for this version of figures * guide: Data Management (main text) finalized for this version of figures * guide: Data Management (secondary text) pending diagram and code sample(s) * guide: add DVC data mgmt technical diagram & dummy sample CLI blocks * guide: update Data Mgmt text * guide: udpate text and 2nd figure (Data Mgmt) * guide: draft 2nd and 3rd figures * guide: rewrite Data Mgmt/ How it works & and Benefits/ Tradeoffs Probably still unfinished... Missing more data versioning info? See HTML comments. * guide: update drafts of Data Mgmt figures 2, 3 * guide: Data Mgmt improvements and hide the benefits list for now * guide: separate from Data Mgmt work Rel. #4042 * Apply suggestions from code review * Merge branch main + * other: links to Remotes guide * install: Remote Storage guide links * start: Remote Storage guide links + * guide: links to Remote Storage page * Restyled by prettier (#4323) Co-authored-by: Restyled.io <[email protected]> --------- Co-authored-by: Dave Berenbaum <[email protected]> Co-authored-by: rogermparent <[email protected]> Co-authored-by: restyled-io[bot] <32688539+restyled-io[bot]@users.noreply.github.com> Co-authored-by: Restyled.io <[email protected]>

* guide: draft structure of Data Mgmt and some updates around the topic in existing docs * guide: full text for draft intro to DM * guide: hide cloud versioning info per #4042 (review) * guide: clarify Data Mgmt parts and add prospective figure titles * guide: add figure drafts to Data Mgmt * guide: SCM->VC (Data Mgmt) * guide: update 2 figs and add 1 more (Data Mgmt) * guide: roll back unrelated changes per #4042 (review) * guide: mention clouds first (DM) and and update fig. 1 per #4042 (review) * guide: flatten DM index per #4042 (review) * guide: udpates to DM/ DV moved from #4053 (review) * guide: add DM/ Data Versioning page per #4042 (comment) * guide: update outdated link * guide: revert more unrelatedly chaqnged files per #4042 (review) * guide: remove unused ref link * guide: DM/ Remote Storage (not just Setup) and and some links from cmd refs and avoid term "data remote" and some admons nearby... * guide: remove a comment * guide: draft for DM/ Remote Storage content * ref: expand config.remote and link to/from Remotes guide * ref: fix remote config file examples * guide: complete Remote Config section and and add Project config section to DM/ DV guide * ref: rewrite remote add and modify Descs * guide: complete list of supported storage types * ref: rewrite remote index page from extracted from #4053 * guide: clarify `remote modify` phrase in in the Remote config section of DM/ Remote Storage * Update content/docs/user-guide/data-management/data-versioning.md * guide: update versioning config per #4058 (review) * guide: don't call remote storage "additional" here (in the DM/ Remote Storage guide) per #4058 (review) Co-authored-by: Dave Berenbaum <[email protected]> * guide: pull -> download (DM/ RS intro) * guide: remove "optional" from Remote Storage nav & title per #4058 (review) * guide: splits and notes around Data Mgmt index page rel. #4042 (comment) * guide: Data Mgmt intro + note updates * guide: draft of all contents + + remove comments * guide: small impros to Data Mgmt in prep for #4042 (review) * guide: rewrite Data Mgmt index in before/after form per #4042 (review) * guide: add draft figure for Data Mgmt * guide: simplify/refocus data mgmt index per #4042 (review) * work around commented header bug * guide: drop DM/ DV page * guide: rewrite DM intro and - hide benefits (for now) - remove codification comment block * guide: use DM table instead of figure for now * guide: rewrite Data Mgmt story * guide: add draft figures to Data Mgmt * guide: simplify Data Mgmt story and benefits * guide: remove unused images (DM) * guide: update Data Mgmt figures (v1) * guide: rewrite text of Data Mgmt index * guide: update Data Mgmt figures * guide: iterate on Data Mgmt again * guide: update Data Mgmt figs * guide: more supporting info about Data Mgmt * guide: update figures (much more concrete) and and matching text updates * guide: edits to How it works (Data Mgmt) * guide: update Data Mgmt figures Rel. #4042 (comment) * guide: emphaisze dataset versions in UG fig 1 Rel. #4042 (comment) * guide: update Data Mgmt figures (with notes), expand img captions, and update text accordingly. * guide: more updates to text and figure styles, esp. to the first half and comment some stuff out (temporary) * guide: update figures and text (Data Mgmt) ... Using a tabs toggle for the 2nd fig. * guide: Data Management text (section 1) finalized for this version of figures * guide: Data Management (main text) finalized for this version of figures * guide: Data Management (secondary text) pending diagram and code sample(s) * guide: add DVC data mgmt technical diagram & dummy sample CLI blocks * guide: update Data Mgmt text * guide: udpate text and 2nd figure (Data Mgmt) * guide: draft 2nd and 3rd figures * guide: rewrite Data Mgmt/ How it works & and Benefits/ Tradeoffs Probably still unfinished... Missing more data versioning info? See HTML comments. * guide: update drafts of Data Mgmt figures 2, 3 * guide: Data Mgmt improvements and hide the benefits list for now * guide: separate from Data Mgmt work Rel. #4042 * Apply suggestions from code review * Merge branch main + * ref: update links from API to Remotes guide * guide: update links around Remote Storage and and other updates to nearby Markdown (e.g. proper admons) * Roll back unrelated changes * Restyled by prettier (#4261) Co-authored-by: Restyled.io <[email protected]> * ref: bring cloud versioning copy edits of import-url from https://github.com/iterative/dvc.org/pull/4260/files#diff-ef95e18c4bd039757695065a23946dc27e28b4727ce07c670cdc096e34dbe3b3 * ref: clarify import-url with cloud versioning per #4142 (review) * ref: updates to import-url --version-aware and update --rev * ref: add import-url --version aware to Synopsis per #4089 (comment) * Restyled by prettier (#4266) Co-authored-by: Restyled.io <[email protected]> * Restyled by prettier (#4322) Co-authored-by: Restyled.io <[email protected]> * Update content/docs/command-reference/remote/modify.md Co-authored-by: Oded Messer <[email protected]> * Update content/docs/command-reference/remote/modify.md Co-authored-by: Oded Messer <[email protected]> * Update content/docs/command-reference/push.md Co-authored-by: Oded Messer <[email protected]> * yarn format-all --------- Co-authored-by: Dave Berenbaum <[email protected]> Co-authored-by: rogermparent <[email protected]> Co-authored-by: restyled-io[bot] <32688539+restyled-io[bot]@users.noreply.github.com> Co-authored-by: Restyled.io <[email protected]> Co-authored-by: Oded Messer <[email protected]>

jorgeorpinel added 10 commits October 13, 2022 02:13

guide: draft structure of Data Mgmt and

7350938

some updates around the topic in existing docs

guide: full text for draft intro to DM

203f6a6

Merge branch 'main' into guide/data-mgmt-flows

90eaa5d

guide: hide cloud versioning info

eb246bb

per #4042 (review)

guide: clarify Data Mgmt parts and

a3687ec

add prospective figure titles

guide: add figure drafts to Data Mgmt

fad0bad

guide: SCM->VC (Data Mgmt)

4e3c3da

guide: update 2 figs and add 1 more (Data Mgmt)

7f02c15

Merge branch 'main' into guide/data-mgmt-flows

f41d16e

Merge branch 'main' into guide/data-mgmt-flows

3a9a045

jorgeorpinel added A: docs Area: user documentation (gatsby-theme-iterative) C: guide Content of /doc/user-guide labels Oct 20, 2022

shcheklein temporarily deployed to dvc-org-guide-data-mgmt-4lzkrz October 20, 2022 21:54 Inactive

jorgeorpinel changed the base branch from main to guide/data-mgmt-flows October 20, 2022 22:09

Merge branch 'main' into guide/data-mgmt-flows

df40521

jorgeorpinel force-pushed the guide/data-mgmt/remote-config branch from 3a9a045 to df40521 Compare October 20, 2022 22:09

shcheklein temporarily deployed to dvc-org-guide-data-mgmt-4lzkrz October 20, 2022 22:10 Inactive

jorgeorpinel added 11 commits October 20, 2022 21:35

Merge branch 'guide/data-mgmt-flows' into guide/data-mgmt/remote-config

adc13ee

guide: roll back unrelated changes

c0b92f1

per #4042 (review)

Merge branch 'guide/data-mgmt-flows' into guide/data-mgmt/remote-config

636872a

guide: mention clouds first (DM) and

c2303c0

and update fig. 1 per #4042 (review)

guide: flatten DM index

62997ab

per #4042 (review)

guide: udpates to DM/ DV

fc74c53

moved from #4053 (review)

guide: add DM/ Data Versioning page

8c40a03

per #4042 (comment)

guide: update outdated link

1a8ca61

guide: revert more unrelatedly chaqnged files

27be87f

per #4042 (review)

guide: remove unused ref link

aaee7af

Merge branch 'guide/data-mgmt-flows' into guide/data-mgmt/remote-config

dd99f21

shcheklein temporarily deployed to dvc-org-guide-data-mgmt-4lzkrz October 22, 2022 07:44 Inactive

jorgeorpinel added 2 commits January 24, 2023 04:28

Merge branch 'main' into guide/data-mgmt-flows

061a918

Merge branch 'guide/data-mgmt-flows' into guide/data-mgmt/remote-config

ac50c94

jorgeorpinel requested a review from a team as a code owner January 24, 2023 09:33

shcheklein temporarily deployed to dvc-org-guide-data-mgmt-mqjkk1 January 24, 2023 09:34 Inactive

jorgeorpinel changed the base branch from guide/data-mgmt-flows to main January 24, 2023 09:39

jorgeorpinel removed the ⌛ status: wait-core-merge Waiting for related product PR merge/release label Jan 24, 2023

jorgeorpinel marked this pull request as draft January 24, 2023 09:40

shcheklein had a problem deploying to dvc-org-guide-data-mgmt-mjuyqx January 24, 2023 09:43 Failure

guide: separate from Data Mgmt work

d781fdd

Rel. #4042

shcheklein temporarily deployed to dvc-org-guide-data-mgmt-mjuyqx January 24, 2023 09:49 Inactive

jorgeorpinel marked this pull request as ready for review January 24, 2023 09:50

shcheklein temporarily deployed to dvc-org-guide-data-mgmt-gihz4u January 24, 2023 10:02 Inactive

guide: remove hidden Storage locations page for now

a8acb25

shcheklein deployed to dvc-org-guide-data-mgmt-gihz4u January 24, 2023 21:22 View deployment

guide: small cleanup of Remote storage page

882170a

shcheklein temporarily deployed to dvc-org-guide-data-mgmt-gihz4u January 24, 2023 21:36 Inactive

jorgeorpinel commented Jan 24, 2023

View reviewed changes

jorgeorpinel merged commit 4d4cbd4 into main Jan 24, 2023

jorgeorpinel deleted the guide/data-mgmt/remote-config branch January 24, 2023 21:40

This was referenced Jan 25, 2023

ref: links to Remote Storage guide, and... #4260

Merged

guide: more links to Remote Storage page #4262

Merged

jorgeorpinel added p0-critical Affects users in a bad way at the moment and removed p1-important Active priorities to deal within next sprints labels Feb 17, 2023

jorgeorpinel added p1-important Active priorities to deal within next sprints and removed p0-critical Affects users in a bad way at the moment labels Mar 16, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

guide: remote storage #4058

guide: remote storage #4058

jorgeorpinel commented Oct 20, 2022 •

edited

Loading

github-actions bot commented Oct 20, 2022 •

edited

Loading

github-actions bot commented Jan 24, 2023 •

edited

Loading

jorgeorpinel Jan 24, 2023

jorgeorpinel commented Jan 24, 2023

guide: remote storage #4058

guide: remote storage #4058

Conversation

jorgeorpinel commented Oct 20, 2022 • edited Loading

github-actions bot commented Oct 20, 2022 • edited Loading

Link Check Report

github-actions bot commented Jan 24, 2023 • edited Loading

Link Check Report

jorgeorpinel Jan 24, 2023

Choose a reason for hiding this comment

jorgeorpinel commented Jan 24, 2023

jorgeorpinel commented Oct 20, 2022 •

edited

Loading

github-actions bot commented Oct 20, 2022 •

edited

Loading

github-actions bot commented Jan 24, 2023 •

edited

Loading