guide: Data Management

[jorgeorpinel]

Goal: Clarify about storage, caching, remotes, and different workflows to handle large data files and dirs DVC's approach to data mgmt early in the guide + possibly concentrate related info from existing docs here.

p.s. closes #1762

• Address first check box in guide: Data Management #2856 (comment).
• Update 1: guide: Data Management #4042 (review)

Main file to review: content/docs/user-guide/data-management/index.md

In review app: https://dvc-org-guide-data-mgmt-yrwkeh.herokuapp.com/doc/user-guide/data-management

---

Next PRs:

• Basic Ops guide: Basic Operations (Data Mgmt) #4053
• Remote storage guide: remote storage #4058

[github-actions]

ab55389

Link Check Report

All 18 links passed!

[dberenbaum]

@shcheklein Are you waiting for me to review this one?

TBH I have some reservations:

• It's long for an index page
• The diagrams are complex
• It ignores cloud versioning
• Both the intro and "Tradeoffs" sections read more like our use cases than our UG

[shcheklein]

@dberenbaum thanks for the feedback. Arguably (@jorgeorpinel can confirm) this PR was one the most complicated in making.

Overall, the intention was to explain somehow to users about the general changes to the way the deal with DVC. In our experience the whole concept of content-addressable storage + a new layer of indirection that comes from Git and codification is not something that would a person expect. It makes a lot of mental effort to understand the implications of this changes.

The intention here was to:

• state a general problem (something that every team solves - e.g. they have to come up with some data conventions, data workflow, etc)
• show an image that would resonate with them (kinda "before") - a messy cloud + people using data directly with AWS CLI, boto, etc
• show an image that shows the same situation with DVC (codified, less mess, different way to access)
• highlight in one paragraph benefits
• here on in the next section start explaining technical details, cache, how exactly storage is organized, etc. (actual UG content, but we need to set the ground).

---

Saying all that, I agree on the points you mentioned. Things that come to my mind:

• Try to shorten the intro (remove the third diagram)
• Try to simplify them - here I would love to hear your take
• Tradeoffs - remove them? Move them into a separate subsection?
• Same for the How it works? Move into a separate page?

Both the intro and "Tradeoffs" sections read more like our use cases than our UG

Yes, they overlap. But we need to set the ground as I mentioned and the biggest difference is that we go technical (e.g. we show CLI commands on diagrams, etc). Anyways, any other idea on the intro?

[dberenbaum]

• Try to shorten the intro (remove the third diagram)

👍

• Try to simplify them - here I would love to hear your take

Are there versions of the diagrams I can modify?

• Tradeoffs - remove them? Move them into a separate subsection?

I would vote to remove this section. If there's anything you think is crucial, can we try to incorporate it into the sections above?

• Same for the How it works? Move into a separate page?

I think it's fine as is if we make the other changes.

[dberenbaum]

Following up with specific suggestions for the diagrams.

Diagram 1

I think this one is pretty good, but I have some minor suggestions:

• scp ssh://...: I guess the idea is to show the raw data comes from somewhere else? Not sure it's worth it and may add more confusion. If you think it's important, we probably need a way to show it outside the S3/"Mass storage" box. Otherwise, I would suggest changing to aws s3 cp ....
• I get the initial impression that the "ML project" workspace is less organized than the "Mass storage" because the "Mass storage" is shorter, but I think we are trying to show the opposite. For example (see a proposed update to "Mass storage" below):
  • training and validation are expanded in "ML project" but collapsed in "Mass storage." Why not expand in "Mass storage" also?
  • There are not that many versions in "Mass storage." Even where there is a training_split_2, there's no corresponding validation_split_2.
  • On the other hand, the Sarah/ directory doesn't correspond to anything from the left. That may help show how the storage becomes a mess, but I think it adds more confusion since it's not clear what it is.

Mass storage
---
dataset.zip
training_split/
  image_1
  ...
  image 1000
training_split_2/
  image_1
  image_3
  ...
validation_split/
  image_1001
  ...
  image_1100
validation_split_2/
  image_2
  image_7
  ...
models/
  model_linear
  model_sarah
  model_final

Diagram 2

Can we abstract away the cloud storage box on the right with something like "Your cloud storage," similar to the "Git hosting" box?

My reasoning:

• We already have detail of how the cache works on the left, and this would help simplify the diagrams.
• This would cover both cache and cloud-versioned remotes (and be more future-proof).
• It emphasizes the indirection and similarity to Git: like Git, it doesn't matter the structure of what you push because you interact via the DVC interface.

Other suggestions for diagram 2:

• Can we use the same title where we have "Mass storage" in diagram 1 and "Storage" in diagram 2? Can we use the same color for them?
• For the "ML project," can we use the same color in both diagrams and use the same basic project structure?
• For the push/pull arrows, can we make them the same for DVC and Git? It seems like there's something different between them since the DVC section mentions DVC, has two arrows, and labels them push and pull. The Git section only has a single bidirectional arrow. Can we make them identical except for one mentioning DVC and the other mentioning Git so it's clear they work similarly?
• Do we need "Git repository v1" at the bottom? It looks like it also applies the DVC cache.

[jorgeorpinel]

long for an index page

I don't think we officially made a difference between index pages and the first page in each section (which sometimes are the first to load - example). But anyway, an easy fix is to move How it works + maybe Tradeoff to a separate page (as suggested by @shcheklein).

diagrams are complex

Try to simplify them... versions of the diagrams I can modify?

We did so many iterations that I was just using pen and paper by the end haha (faster). The designer produced them from there.

ignores cloud versioning

Not entirely @dberenbaum (See the 💡 Remote storage admon) but I did avoid going into that in detail since it's closer to backing up/ sharing data (having it's own guide page); And trying to include it further complicated the doc.

intro and "Tradeoffs" sections read more like our use cases

The main difference is the level of technicality (ditto) bu another option to consider, esp. if we manage to produce less complex diagrams, is to move the intro + maybe Benefits to a Use Case and start this page with the content of How it works (and maybe some of the current diagrams).

[jorgeorpinel]

On the 1st diagram:

the "ML project" workspace is less organized

Initially there were file counts next to the directory sizes to avoid opening trees. Maybe reinstating that or skipping that aspect altogether is better than expanding dirs everywhere (too long).

not that many versions in "Mass storage."

If you think adding a few more makes it more realistic (a problem readers can relate to) then I'd agree.

the Sarah/ directory ... may help show how the storage becomes a mess

Direct storage access leads to different people doing whatever they want/can (messy, insecure, no trace, etc).

On the 2nd diagram:

abstract away the cloud storage box on the right

Good idea! Would def. make it simpler and better cover cloud versioning. It's only purpose was to show more than one version controlled by DVC (while only one is in the left side) but a) it's kind of hard to get that without analyzing the details and b) maybe that can be expressed in a different way, like stacking Git versions on the left side.

"Git repository v1" at the bottom... also applies the DVC cache

Not sure I get how it applies to the cache.

[jorgeorpinel]

Q: Is there a subset of actions from above (except diagram updates) I can take to make this mergeable?

[dberenbaum]

Q: Is there a subset of actions from above (except diagram updates) I can take to make this mergeable?

• Try to shorten the intro (remove the third diagram)

• Tradeoffs - remove them?

I'd like to make those changes and update the other diagrams to make it mergeable.

Here are some proposed polishing of the diagrams that incorporate my feedback above:

Diagram 1

Diagram 2

After spending some time on diagram 2, I think we should drop all the cache/hash details. I think we are trying to do too much in one diagram (showing the value prop and how it works), and dropping those details stresses the simplification and indirection that DVC provides.

If we want a diagram for how it works, I would put it in the next section. Here's one idea for that diagram (not sure it's needed since it's explained well in that section already):

Source: https://miro.com/app/board/uXjVP0xtE7A=/ (I know it doesn't help you @jorgeorpinel 😉 )

[shcheklein]

I can do the diagrams (working on it). I need a better representation of a messy mass storage atm.

[shcheklein]

The first diagram, feedback @dberenbaum @jorgeorpinel

[dberenbaum]

@shcheklein I think it does a better job of showing a messy mass storage, although I worry that we focus so much on creating a messy storage that it becomes incomprehensible to readers.

I'd like to see how it connects to diagram 2, like a before and after of the same project.

[shcheklein]

although I worry that we focus so much on creating a messy storage that it becomes incomprehensible to readers.

yep, but that's the whole point of this - I want this to resonate with some pain points. I agree that it gets complicated though. I don't know a better way for now.

What I definitely don't like is when it gets too simple to the point of not delivering the message at all, or when it's artificial and you "don't trust" it, etc.

May be we can do a completely different approach, I don't know.

One more point here to remember - this is User Guide, it's fine for it to be more technical / detail-oriented.

[dberenbaum]

@shcheklein Do you plan to get back to updating the 2nd diagram? Do you want me to unblock and merge as is? Do you want me to take it over?

[shcheklein]

@dberenbaum I plan to get back to it. Can't promise any deadlines though. If you have some ideas for the second diagram - I can try them.

[dberenbaum]

Not sure this needs its own page, but just noticed it is not included in the sidebar.

[shcheklein]

Clearly we don't have enough capacity for this. Closing and we can get back to it later.