dvcfs: initial docs #3932

skshetry · 2022-09-09T15:10:30Z

Fixes partially #3927.

First draft of dvcfs.

skshetry · 2022-09-09T15:11:37Z

content/docs/api-reference/dvcfs.md

+DVC-tracked into a local directory "data". Similarly, DVC might fetch files from
+remote if they don't exist in the cache.
+
+## API Reference


I am not really seeing much value from "API reference" right now compared to Scenarios/Recipes that we can cover.

Goes back to #3927 (comment). If we have the API ref auto-generated then I agree. Or do you suggest skipping it completely and let people/IDEs check docstrings?

fsspec is supposed to be a documented API that this library implements, we don't want to repeat everything here. Also, it makes sense to start small, with examples. I hope a lot of things here will be self descriptive through examples.

content/docs/api-reference/dvcfs.md

github-actions · 2022-09-09T15:21:42Z

Link Check Report

content/docs/api-reference/dvcfilesystem.md
- PASS: https://filesystem-spec.readthedocs.io/ (200)
- PASS: /doc/command-reference/remote/add#supported-storage-types = https://dvc-org-dvcfs-zfsolvaiaztew0ke.herokuapp.com/doc/command-reference/remote/add#supported-storage-types (200)
- PASS: https://git-scm.com/docs/revisions (200)
- PASS: https://filesystem-spec.readthedocs.io/ (200)
- PASS: https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem (200)
- PASS: /doc/command-reference/exp/run#-n = https://dvc-org-dvcfs-zfsolvaiaztew0ke.herokuapp.com/doc/command-reference/exp/run#-n (200)

All 6 links passed!

dberenbaum · 2022-09-09T16:57:14Z

@skshetry Is it ready for initial review?

skshetry · 2022-09-09T17:05:11Z

@dberenbaum, I'd say yes. This is a bare-bones version, as I wanted to start simple. And there are open questions regarding docs organization and API reference.

I think in these docs, I am going to approach this with more code examples/recipes instead of API references.

Here's the PR: iterative/dvc#8277 that this is based on (I made some changes to the constructor of the filesystem).

shcheklein

I like the approach. I also don't see a benefit of repeating the whole fsspec (even though it doesn't have proper docs at all). We might need to iterate of the text on the first two paragraphs and merge it.

content/docs/api-reference/dvcfs.md

shcheklein · 2022-09-09T23:24:31Z

content/docs/api-reference/dvcfs.md

+
+DvcFileSystem provides a single view of all the files/directories in your
+repository, be it Git-tracked or DVC-tracked, or untracked (in case of a local
+repository). DvcFileSystem is smart to reuse the files in your cache directory


Internally it avoids downloading DVC-tracked files unless it's needed. It's using DVC cache [make it abbr] to avoid downloading objects every time and support streaming for supported remote storages [link] for operations like ...

content/docs/api-reference/dvcfs.md

shcheklein · 2022-09-09T23:26:34Z

content/docs/api-reference/dvcfs.md

+
+## Examples
+
+### Listing all DVC-tracked files


suggestion: make title look like command/calls - ls: list DVC-tracked files

The snippet here uses find, not ls. The title should probably reflect that

Like @daavoo said, it's equivalent to ls --recursive, so I don't want to force commands in the title. I do mention that after the snippets now. :)

shcheklein · 2022-09-09T23:27:13Z

content/docs/api-reference/dvcfs.md

+### Listing all DVC-tracked files
+
+```py
+>>> fs = DvcFileSystem("https://github.com/iterative/example-get-started.git", rev="main")


may be worth mentioning in those examples important aspects of their behavior - e.g. it doesn't download files, streams, etc. Can be brief

content/docs/api-reference/dvcfs.md

dberenbaum · 2022-09-11T10:55:23Z

content/docs/api-reference/dvcfs.md

+## API Reference
+
+As DvcFileSystem is based on [fsspec](https://filesystem-spec.readthedocs.io/),
+it is compatible with most of the APIs that it offers. Please check the fsspec's


Suggested change

it is compatible with most of the APIs that it offers. Please check the fsspec's

it is compatible with most of the read-only methods that it offers. Please check the fsspec's

It's not the API that is read-only, but the filesystem and its operation in general. I have also struggled to explain this, but tbh I don't think we need to mention anything other than "... is a read-only filesystem".

I think it's better to even skip it and raise a EROFS.

What is an EROFS?

I must be missing your point here because I don't see what's confusing or ambiguous. Is dvcfs compatible with the read-only methods of https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem?

EROFS is an errno which is raised for "Read Only FileSystem". What I am trying to say is that the filesystem in general is read-only, not the methods. Take fs.open() for example. It works with mode="rb" but mode="wb" raises EROFS. So I am trying to distinguish between the method open vs the operations: writing/reading, etc.

dberenbaum · 2022-09-11T11:01:53Z

content/docs/api-reference/dvcfs.md

@@ -0,0 +1,66 @@
+# DvcFileSystem
+
+DvcFileSystem provides a pythonic file interface (aka


Maybe it can be further simplified to stress the main points that it's read-only and fsspec comapitable. I think it should start with something like, "DVCFileSystem is a read-only fsspec-compatible interface." Then you could go on to describe the typical operations and everything else.

Initially I wrote like that, but I later thought that the end users does not care about fsspec or need to care about it to use it.

The read-only part is important

Initially I wrote like that, but I later thought that the end users does not care about fsspec or need to care about it to use it.

Good point, but if we are not documenting the available methods, we need to point out clearly that it implements the read-only methods available in fsspec.

If you want to first focus on it being a filesytem-like interface and read-only, you could say something like: "DvcFileSystem provides read-only operations like ls, du, glob, get, download, etc. for a DVC repo. The API implements most read-only methods of fsspec."

the end users does not care about fsspec or need to care about it to use it.

So why do we even mention it?

It is an fsspec-based filesystem. We do need to mention it, I just don't think we should lead with that in the intro. 🙂

Sounds good. Please see #3932 (comment) below. Thanks

content/docs/api-reference/dvcfs.md

content/docs/sidebar.json

jorgeorpinel · 2022-09-14T07:12:37Z

Many comments already so I'll just summarize some thoughts here:

Nothing against the example-based approach but
- It's completely different to the current functions we have listed; Should we group those under dvc.api in the nav and use dvc.fs for this one?
- Are the examples exhaustive or comprehensive? (How do we decide what to cover?)
In general the exact relation with fsspec is currently vague. Is it literally fsspec? The link at the bottom goes to fsspec.spec.AbstractFileSystem, is that what we implement? This link seems a better intro doc on their side BTW: https://filesystem-spec.readthedocs.io/en/latest/usage.html#use-a-file-system

skshetry · 2022-09-15T02:30:22Z

It's completely different to the current functions we have listed; Should we group those under dvc.api in the nav and use dvc.fs for this one?

It's now under dvc.api.

Are the examples exhaustive or comprehensive? (How do we decide what to cover?)

I don't think we have to be strict about it, we can always update it and iterate.

In general the exact relation with fsspec is currently vague. Is it literally fsspec? The link at the bottom goes to fsspec.spec.AbstractFileSystem, is that what we implement?

yes, fsspec provides a standard interface, "dvcfs" implements it.

content/docs/api-reference/dvcfilesystem.md

* 'dvcfs' of github.com:iterative/dvc.org: Update content/docs/api-reference/dvcfilesystem.md

dberenbaum

It makes sense to start small with examples for this PR, but can we hold off closing #3927 until have a reference of supported methods either in the docs or linked from the docs (to an auto-generated docs site)? Compare to the s3fs docs and gcsfs docs. Both of those document the full API of methods they support. Otherwise, we are leaving users to either look through the codebase (without pointing to where they should look) or go to the fsspec AbstractFileSystem docs and use trial and error to find which methods are implemented.

skshetry · 2022-09-19T03:11:26Z

@jorgeorpinel @shcheklein, I am leaving for a vacation next week, so it'd be great if you could review them, I can iterate on these docs this week.

shcheklein · 2022-09-19T03:31:33Z

@skshetry I see that @dberenbaum has approved it. Are there unresolved things / any blockers / you need help with something?

skshetry · 2022-09-19T03:36:36Z

I can always get more feedback. 🙂 Is it fine to merge this then?

shcheklein · 2022-09-19T03:45:57Z

Yes, I think so, one approval should be enough.

What do you think about keeping the ticket open like Dave (and I guess Jorge suggested)?

skshetry · 2022-09-19T03:47:43Z

What do you think about keeping the ticket open like Dave (and I guess Jorge suggested)?

Done, thanks.

github-actions · 2022-09-19T05:10:48Z

98fe9aa

Link Check Report

There were no links to check!

jorgeorpinel

Just some suggested follow-ups post-merge, concentrated on the intro (the subsections are pretty good I think 👍🏼 -- may just need double checking grammar, etc.)

jorgeorpinel · 2022-10-18T06:40:19Z

content/docs/api-reference/dvcfilesystem.md

+DVCFileSystem provides a pythonic file interface (
+[fsspec-compatible](https://filesystem-spec.readthedocs.io/)) for a DVC repo. It
+is a read-only filesystem, hence it does not support any write operations, like
+`put_file`, `cp`, `rm`, `mv`, `mkdir` etc.


💅🏼

Suggested change

DVCFileSystem provides a pythonic file interface (

[fsspec-compatible](https://filesystem-spec.readthedocs.io/)) for a DVC repo. It

is a read-only filesystem, hence it does not support any write operations, like

`put_file`, `cp`, `rm`, `mv`, `mkdir` etc.

`DVCFileSystem` is a Python class that provides a file interface for <abbr>DVC repositories</abbr>

(compatible with [fsspec](https://filesystem-spec.readthedocs.io/)). It

is read-only, hence it does not support any write operations like

`put_file`, `cp`, `rm`, `mv`, `mkdir` etc.

jorgeorpinel · 2022-10-18T06:43:53Z

content/docs/api-reference/dvcfilesystem.md

+DVCFileSystem provides a unified view of all the files/directories in your
+repository, be it Git-tracked or DVC-tracked, or untracked (in case of a local
+repository). It can reuse the files in DVC <abbr>cache</abbr> and can otherwise
+stream from
+[supported remote storage](/doc/command-reference/remote/add#supported-storage-types).


Suggested change

DVCFileSystem provides a unified view of all the files/directories in your

repository, be it Git-tracked or DVC-tracked, or untracked (in case of a local

repository). It can reuse the files in DVC <abbr>cache</abbr> and can otherwise

stream from

[supported remote storage](/doc/command-reference/remote/add#supported-storage-types).

`DVCFileSystem` objects construct a unified view of all the files and directories in your

repo, be it Git-tracked, DVC-tracked, or untracked (in the case of local

repos). It can reuse files in the <abbr>DVC cache</abbr>, and it can otherwise

stream from [supported remote storage].

[supported remote storage]: /doc/command-reference/remote/add#supported-storage-types

It can reuse files in the DVC cache

What does this mean though? Reuse them for what purpose?

jorgeorpinel · 2022-10-18T06:45:12Z

content/docs/api-reference/dvcfilesystem.md

+```py
+>>> from dvc.api import DVCFileSystem
+# opening a local repository
+>>> fs = DVCFileSystem("/path/to/local/repository")
+# opening a remote repository
+>>> url = "https://github.com/iterative/example-get-started.git"
+>>> fs = DVCFileSystem(url, rev="main")


💅🏼

Suggested change

```py

>>> from dvc.api import DVCFileSystem

# opening a local repository

>>> fs = DVCFileSystem("/path/to/local/repository")

# opening a remote repository

>>> url = "https://github.com/iterative/example-get-started.git"

>>> fs = DVCFileSystem(url, rev="main")

```py

from dvc.api import DVCFileSystem

# Opening a local repository

fs = DVCFileSystem("/path/to/local/repository")

# Opening a remote repository

url = "https://github.com/iterative/example-get-started.git"

fs = DVCFileSystem(url, rev="main")

Applies to other py code blocks.

jorgeorpinel · 2022-10-18T06:51:38Z

content/docs/api-reference/dvcfilesystem.md

+The optional positional argument can be a URL or a local path to the DVC
+project. If unspecified, the DVC project in current working directory is used.


Does it work with any DVC project or does it have to be a Git repo? (We currently use the term "repo" specifically in a few places.)

jorgeorpinel · 2022-10-18T06:52:20Z

content/docs/api-reference/dvcfilesystem.md

+The optional positional argument can be a URL or a local path to the DVC
+project. If unspecified, the DVC project in current working directory is used.
+
+The optional `rev` argument can be passed to open a filesystem from a certain
+Git commit (any [revision](https://git-scm.com/docs/revisions) such as a branch
+or a tag name, a commit hash, or an [experiment name]).
+
+[experiment name]: /doc/command-reference/exp/run#-n


Would it be better to put the class signature up somewhere? With param names, etc. (may need to make a mixed definition combined with FileSystem)

jorgeorpinel · 2022-10-18T06:56:19Z

content/docs/api-reference/dvcfilesystem.md

+## API Reference
+
+As DVCFileSystem is based on [fsspec](https://filesystem-spec.readthedocs.io/),
+it is compatible with most of the APIs that it offers. Please check the fsspec's
+[API Reference](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem)
+for more details.


Last note: we could use terms "properties" and "methods" instead in here, to clarify we mean the reference of this specific class/object, not some larger API.

p.s. we could put this section earlier too, probably before the example sections, even as a note (<admon> or not).

dvcfs: initial docs

564ffdc

skshetry self-assigned this Sep 9, 2022

shcheklein temporarily deployed to dvc-org-dvcfs-zfsolvaiaztew0ke September 9, 2022 15:10 Inactive

skshetry commented Sep 9, 2022

View reviewed changes

content/docs/api-reference/dvcfs.md Outdated Show resolved Hide resolved

skshetry commented Sep 9, 2022

View reviewed changes

content/docs/api-reference/dvcfs.md Outdated Show resolved Hide resolved

shcheklein reviewed Sep 9, 2022

View reviewed changes

content/docs/api-reference/dvcfs.md Outdated Show resolved Hide resolved

shcheklein reviewed Sep 9, 2022

View reviewed changes

content/docs/api-reference/dvcfs.md Outdated Show resolved Hide resolved

shcheklein reviewed Sep 9, 2022

View reviewed changes

content/docs/api-reference/dvcfs.md Outdated Show resolved Hide resolved

shcheklein reviewed Sep 9, 2022

View reviewed changes

content/docs/api-reference/dvcfs.md Outdated Show resolved Hide resolved

shcheklein reviewed Sep 9, 2022

View reviewed changes

dberenbaum reviewed Sep 11, 2022

View reviewed changes

content/docs/api-reference/dvcfs.md Outdated Show resolved Hide resolved

dberenbaum reviewed Sep 11, 2022

View reviewed changes

daavoo reviewed Sep 12, 2022

View reviewed changes

content/docs/api-reference/dvcfs.md Outdated Show resolved Hide resolved

jorgeorpinel reviewed Sep 14, 2022

View reviewed changes

content/docs/sidebar.json Outdated Show resolved Hide resolved

add couple more scenarios, rewrite intro

9c7e41a

shcheklein temporarily deployed to dvc-org-dvcfs-zfsolvaiaztew0ke September 15, 2022 06:52 Inactive

rename dvcfilesystem

ad8bb56

shcheklein temporarily deployed to dvc-org-dvcfs-zfsolvaiaztew0ke September 15, 2022 07:02 Inactive

skshetry commented Sep 15, 2022

View reviewed changes

content/docs/api-reference/dvcfilesystem.md Outdated Show resolved Hide resolved

Update content/docs/api-reference/dvcfilesystem.md

0ad0da5

shcheklein temporarily deployed to dvc-org-dvcfs-zfsolvaiaztew0ke September 15, 2022 12:58 Inactive

skshetry mentioned this pull request Sep 16, 2022

dvcfs: rename DvcFileSystem to DVCFileSystem iterative/dvc#8307

Merged

skshetry added 2 commits September 16, 2022 19:02

rename to DVCFileSystem

d18e2cc

Merge branch 'dvcfs' of github.com:iterative/dvc.org into dvcfs

9751264

* 'dvcfs' of github.com:iterative/dvc.org: Update content/docs/api-reference/dvcfilesystem.md

shcheklein temporarily deployed to dvc-org-dvcfs-zfsolvaiaztew0ke September 16, 2022 13:18 Inactive

dberenbaum approved these changes Sep 16, 2022

View reviewed changes

skshetry merged commit 98fe9aa into main Sep 19, 2022

skshetry deleted the dvcfs branch September 19, 2022 05:01

dberenbaum mentioned this pull request Sep 19, 2022

Document dvcfs API #3927

Closed

jorgeorpinel reviewed Oct 18, 2022

View reviewed changes

jorgeorpinel mentioned this pull request Oct 18, 2022

docs: dvcfs: use read_bytes/read_text instead of cat_file #3990

Merged

jorgeorpinel added A: docs Area: user documentation (gatsby-theme-iterative) C: ref Content of /doc/*-reference labels Oct 18, 2022

	it is compatible with most of the APIs that it offers. Please check the fsspec's
	it is compatible with most of the read-only methods that it offers. Please check the fsspec's

		@@ -0,0 +1,66 @@
		# DvcFileSystem

		DvcFileSystem provides a pythonic file interface (aka

		The optional positional argument can be a URL or a local path to the DVC
		project. If unspecified, the DVC project in current working directory is used.

dvcfs: initial docs #3932

dvcfs: initial docs #3932

Conversation

skshetry commented Sep 9, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Sep 9, 2022 • edited Loading

Link Check Report

dberenbaum commented Sep 9, 2022

skshetry commented Sep 9, 2022

shcheklein left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

skshetry Sep 15, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

skshetry Sep 14, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jorgeorpinel commented Sep 14, 2022

skshetry commented Sep 15, 2022

dberenbaum left a comment

Choose a reason for hiding this comment

skshetry commented Sep 19, 2022

shcheklein commented Sep 19, 2022

skshetry commented Sep 19, 2022

shcheklein commented Sep 19, 2022

skshetry commented Sep 19, 2022

github-actions bot commented Sep 19, 2022

Link Check Report

jorgeorpinel left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jorgeorpinel Oct 18, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jorgeorpinel Oct 18, 2022 • edited Loading

Choose a reason for hiding this comment

skshetry commented Sep 9, 2022 •

edited

Loading

github-actions bot commented Sep 9, 2022 •

edited

Loading

skshetry Sep 15, 2022 •

edited

Loading

skshetry Sep 14, 2022 •

edited

Loading

jorgeorpinel Oct 18, 2022 •

edited

Loading

jorgeorpinel Oct 18, 2022 •

edited

Loading