Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ref/remote: tabify & crossref config #3299

Closed
wants to merge 1 commit into from
Closed

ref/remote: tabify & crossref config #3299

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
113 changes: 54 additions & 59 deletions content/docs/command-reference/remote/add.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,9 +88,9 @@ DVC will determine the [type of remote](#supported-storage-types) based on the

The following are the types of remote storage (protocols) supported:

<details>
<toggle>

### Click for Amazon S3
<tab title="Amazon S3">

> 💡 Before adding an S3 remote, be sure to
> [Create a Bucket](https://docs.aws.amazon.com/AmazonS3/latest/gsg/CreatingABucket.html).
Expand All @@ -109,13 +109,13 @@ Make sure you have the following permissions enabled: `s3:ListBucket`,
methods that are performed by DVC (`list_objects_v2` or `list_objects`,
`head_object`, `upload_file`, `download_file`, `delete_object`, `copy`).

> See `dvc remote modify` for a full list of S3 parameters.
> See
> [`dvc remote modify`](/doc/command-reference/remote/modify?tab=Amazon-S3#available-parameters-per-storage-type)
> for a full list of S3 parameters.

</details>
</tab>

<details>

### Click for S3-compatible storage
<tab title="S3-compatible storage">

For object storage that supports an S3-compatible API (e.g.
[Minio](https://min.io/),
Expand All @@ -134,16 +134,15 @@ $ dvc remote modify myremote endpointurl \
By default, DVC authenticates using your AWS CLI
[configuration](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html)
(if set). This uses the default AWS credentials file. To use a custom
authentication method, use the parameters described in `dvc remote modify`.
authentication method, use the parameters described in
[`dvc remote modify`](/doc/command-reference/remote/modify?tab=S3-compatible-storage#available-parameters-per-storage-type).

Any other S3 parameter can also be set for S3-compatible storage. Whether
they're effective depends on each storage platform.

</details>

<details>
</tab>

### Click for Microsoft Azure Blob Storage
<tab title="Microsoft Azure Blob Storage">

```dvc
$ dvc remote add -d myremote azure://mycontainer/path
Expand All @@ -158,14 +157,13 @@ a Microsoft application.
https://docs.microsoft.com/en-us/python/api/azure-identity/azure.identity.defaultazurecredential

To use a custom authentication method, use the parameters described in
`dvc remote modify`. See some
[`dvc remote modify`](/doc/command-reference/remote/modify?tab=Microsoft-Azure-Blob-Storage#available-parameters-per-storage-type).
See some
[examples](/doc/command-reference/remote/modify#example-some-azure-authentication-methods).

</details>
</tab>

<details>

### Click for Google Drive
<tab title="Google Drive">

To start using a GDrive remote, first add it with a
[valid URL format](/doc/user-guide/setup-google-drive-remote#url-format). Then
Expand All @@ -183,7 +181,9 @@ Go to the following link in your browser:
Enter verification code: # <- enter resulting code
```

See `dvc remote modify` for a list of other GDrive parameters, or
See
[`dvc remote modify`](/doc/command-reference/remote/modify?tab=Google-Drive#available-parameters-per-storage-type)
for a list of other GDrive parameters, or
[Set up a Google Drive DVC Remote](/doc/user-guide/setup-google-drive-remote)
for a full guide on using Google Drive as DVC remote storage.

Expand All @@ -195,11 +195,9 @@ modified.

> Please note our [Privacy Policy (Google APIs)](/doc/user-guide/privacy).

</details>

<details>
</tab>

### Click for Google Cloud Storage
<tab title="Google Cloud Storage">

> 💡 Before adding a GC Storage remote, be sure to
> [Create a storage bucket](https://cloud.google.com/storage/docs/creating-buckets).
Expand All @@ -211,17 +209,16 @@ $ dvc remote add -d myremote gs://mybucket/path
By default, DVC expects your GCP CLI is already
[configured](https://cloud.google.com/sdk/docs/authorizing). DVC will be using
default GCP key file to access Google Cloud Storage. To override some of these
parameters, use the parameters described in `dvc remote modify`.
parameters, use the parameters described in
[`dvc remote modify`](/doc/command-reference/remote/modify?tab=Google-Cloud-Storage#available-parameters-per-storage-type).

> Make sure to run `gcloud auth application-default login` unless you use
> `GOOGLE_APPLICATION_CREDENTIALS` and/or service account, or other ways to
> authenticate. See details [here](https://stackoverflow.com/a/53307505/298182).

</details>

<details>
</tab>

### Click for Aliyun OSS
<tab title="Aliyun OSS">

First you need to set up OSS storage on Aliyun Cloud. Then, use an S3 style URL
for OSS storage, and configure the
Expand All @@ -233,8 +230,8 @@ $ dvc remote modify myremote oss_endpoint endpoint
```

To set key id, key secret and endpoint (or any other OSS parameter), use
`dvc remote modify` as show below. Use the `--local` option to avoid committing
your secrets to Git:
[`dvc remote modify`](/doc/command-reference/remote/modify?tab=Aliyun-OSS#available-parameters-per-storage-type)
as show below. Use the `--local` option to avoid committing your secrets to Git:

```dvc
$ dvc remote modify --local myremote oss_key_id 'mykey'
Expand All @@ -259,29 +256,27 @@ $ export OSS_ACCESS_KEY_SECRET='mysecret'
> Uses default key id and key secret when they are not given, which gives read
> access to public read bucket and public bucket.

</details>
</tab>

<details>

### Click for SSH
<tab title="SSH">

```dvc
$ dvc remote add -d myremote ssh://[email protected]/path
```

> See `dvc remote modify` for a full list of SSH parameters.
> See
> [`dvc remote modify`](/doc/command-reference/remote/modify?tab=SSH#available-parameters-per-storage-type)
> for a full list of SSH parameters.

⚠️ DVC requires both SSH and SFTP access to work with remote SSH locations.
Please check that you are able to connect both ways with tools like `ssh` and
`sftp` (GNU/Linux).

> Note that the server's SFTP root might differ from its physical root (`/`).

</details>

<details>
</tab>

### Click for HDFS
<tab title="HDFS">

⚠️ Using HDFS with a Hadoop cluster might require additional setup. Our
assumption is that the client is set up to use it. Specifically, [`libhdfs`]
Expand All @@ -297,13 +292,13 @@ API. Read more about it by expanding the WebHDFS section below.
$ dvc remote add -d myremote hdfs://[email protected]/path
```

> See `dvc remote modify` for a full list of HDFS parameters.

</details>
> See
> [`dvc remote modify`](/doc/command-reference/remote/modify?tab=HDFS#available-parameters-per-storage-type)
> for a full list of HDFS parameters.

<details>
</tab>

### Click for WebHDFS
<tab title="WebHDFS">

⚠️ Using WebHDFS requires to enable REST API access in the cluster: set the
config property `dfs.webhdfs.enabled` to `true` in `hdfs-site.xml`.
Expand All @@ -325,25 +320,25 @@ active kerberos session.
> `token` contains sensitive user info. Therefore, it's safer to add it with the
> `--local` option, so it's written to a Git-ignored config file.

> See `dvc remote modify` for a full list of WebHDFS parameters.
> See
> [`dvc remote modify`](/doc/command-reference/remote/modify?tab=WebHDFS#available-parameters-per-storage-type)
> for a full list of WebHDFS parameters.

</details>
</tab>

<details>

### Click for HTTP
<tab title="HTTP">

```dvc
$ dvc remote add -d myremote https://example.com/path
```

> See `dvc remote modify` for a full list of HTTP parameters.

</details>
> See
> [`dvc remote modify`](/doc/command-reference/remote/modify?tab=HTTP#available-parameters-per-storage-type)
> for a full list of HTTP parameters.

<details>
</tab>

### Click for WebDAV
<tab title="WebDAV">

```dvc
$ dvc remote add -d myremote \
Expand All @@ -358,13 +353,13 @@ $ dvc remote add -d myremote \
webdavs://example.com/owncloud/remote.php/dav/files/myuser
```

> See `dvc remote modify` for a full list of WebDAV parameters.

</details>
> See
> [`dvc remote modify`](/doc/command-reference/remote/modify?tab=WebDAV#available-parameters-per-storage-type)
> for a full list of WebDAV parameters.

<details>
</tab>

### Click for local remote
<tab title="local remote">

A "local remote" is a directory in the machine's file system. Not to be confused
with the `--local` option of `dvc remote` (and other config) commands!
Expand Down Expand Up @@ -402,7 +397,7 @@ $ cat .dvc/config
> Note that `../dvcstore` has been resolved relative to the `.dvc/` dir,
> resulting in `../../dvcstore`.

</details>
</tab>

## Example: Customize an S3 remote

Expand Down
Loading