guide: extract remote add/modify details from cmd ref.

[jorgeorpinel]

Currently most of the information about remote storage is in the command reference, mainly under the remote add/modify commands (which have expandable details sections for each supported remote type. We can leave some basics in there (cmd refs should be self-contained), but the details need to be in the User Guide instead. Probably as part of #2856

Some related tickets that could potentially be included in this refactoring:

• keep in mind ref: better prioritize and highlight common remote options #1699
• guide: working with SSH remote storage #3832
• guide: using NFS as a remote storage #103
• ref: clarify about obfuscated remote file structure (same as cache) #1762
• + Cover Sharing Data/Models ? (See [spike] cases: address Sharing Data #3186)

This is may be related to #340, as remote definitions are part of the configuration of DVC. But we probably still want a separate guide for this as the remote storage feature goes beyond configuration.

[daavoo]

During support dutty, I usually see a lot of questions related to setting up credentials for remotes.

Perhaps we could try to have guides like the one for Google Drive (with screenshots, etc.) for other popular remotes? (no stats to support but AWS and Azure appear to be the most commonly asked about)

[jorgeorpinel]

Perhaps we could try to have guides like the one for Google Drive (with screenshots, etc.) for other popular remotes?

That would be quite nice but it takes lots of effort to produce and maintain. That guide was generously contributed (and already updated once), yet is probably already outdated as GCP changes their UI and even API regularly.

We def. want to un-bury this info onto the User Guide, but avoid documenting 3rd-party tools whenever possible. That said pointing out the relevant config fields for each remote type is probably very valuable for users, so they don't have to investigate the entire (huge) spec. on storage provider docs.

[daavoo]

That would be quite nice but it takes lots of effort to produce and maintain.

I think it would be worth it, even if the screenshots become eventually outdated. The current page in the user guide (https://dvc.org/doc/user-guide/setup-google-drive-remote) is actually very popular. According to plausible stats, it is more visited than remote modify cmd ref.

We already have some blog posts with similar content for other providers :

• https://dvc.org/blog/azure-remotes-in-dvc
• https://dvc.org/blog/aws-remotes-in-dvc

Maybe we could consolidate all of this guides in a single page/section, instead of having 1 provider in user-guide and a couple (and more coming #3560) as blog posts?

[jorgeorpinel]

Again, I agree it would be great but it's not about being worth-it: we just don't have the capacity to maintain that now, unless the core team wants to own that on top of the cmd ref.

According to plausible stats, it is more visited than remote modify cmd ref.

They're about the same since people spend more time in the latter. And there's also the remote/add ref., which is about twice as popular as the GDrive guide. Still good relative numbers for a specific remote guide, I agree.

We already have some blog posts with similar content

And more are in the works I think. So yes, the blog may resolve part of this at no future cost since we don't maintain blog posts. At some point they become totally dated though (years out I guess).

consolidate all of this guides in a single page/section

Yep. That's the idea for this issue (but not nearly as detailed as the GDrive guide). BTW this (extracting from the cmd ref) should help SEO too.

[jorgeorpinel]

UPDATE: While Is till don't think we can handle tutorial-form remote guides for every remote type, we can have a few (on top of blog posts) for the most requested ones via support. I.e. S3 #3832

[jorgeorpinel]

Hi @dberenbaum. While #103 (comment) is resolved/decided, would it be OK to give #3832 a go? It's a task here and marked p1.

p.s. I see most other storage types have been moved out of the ref, nice!

[dberenbaum]

If you want to try to make https://dvc.org/doc/user-guide/data-management/remote-storage/ssh into more of a true guide, please feel free @jorgeorpinel! Thanks!

[jorgeorpinel]

This seems like it's finished, probably. Esp. after #4384 is merged.