-
Notifications
You must be signed in to change notification settings - Fork 394
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
remote: use consistent order and terminology for remote types
per #833 (review)
- Loading branch information
1 parent
d9ab97f
commit 63a4ac9
Showing
17 changed files
with
221 additions
and
220 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -45,10 +45,11 @@ DVC supports several types of (local or) remote locations (protocols): | |
| | `hdfs` | HDFS | `hdfs://[email protected]/path/to/data.csv` | | ||
| | `http` | HTTP to file | `https://example.com/path/to/data.csv` | | ||
|
|
||
| > Depending on the remote locations type you plan to download data from you | ||
| > might need to specify one of the optional dependencies: `[s3]`, `[ssh]`, | ||
| > `[gs]`, `[azure]`, `[gdrive]`, and `[oss]` (or `[all]` to include them all) | ||
| > when [installing DVC](/doc/install) with `pip`. | ||
| > If you installed DVC via `pip` and plan to use cloud services as remote | ||
| > storage, you might need to install these optional dependencies: `[s3]`, | ||
| > `[azure]`, `[gdrive]`, `[gs]`, `[oss]`, `[ssh]`. Alternatively, use `[all]` to | ||
| > include them all. The command should look like this: `pip install "dvc[s3]"`. | ||
| > (This example installs `boto3` library along with DVC to support S3 storage.) | ||
| Another way to understand the `dvc get-url` command is as a tool for downloading | ||
| data files. | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -23,20 +23,20 @@ positional arguments: | |
|
|
||
| ## Description | ||
|
|
||
| `name` and `url` are required. `url` specifies a location to store your data. It | ||
| can be an SSH, S3 path, Azure, Google Drive path, Google Cloud path, Aliyun OSS, | ||
| local directory, etc. (See all the supported remote storage types in the | ||
| examples below.) If `url` is a local relative path, it will be resolved relative | ||
| to the current working directory but saved **relative to the config file | ||
| location** (see LOCAL example below). Whenever possible DVC will create a remote | ||
| directory if it doesn't exists yet. It won't create an S3 bucket though and will | ||
| rely on default access settings. | ||
|
|
||
| > If you installed DVC via `pip`, depending on the remote storage type you plan | ||
| > to use you might need to install optional dependencies: `[s3]`, `[ssh]`, | ||
| > `[gs]`, `[azure]`, `[gdrive]`, and `[oss]`; or `[all]` to include them all. | ||
| > The command should look like this: `pip install "dvc[s3]"`. This installs | ||
| > `boto3` library along with DVC to support Amazon S3 storage. | ||
| `name` and `url` are required. `url` specifies a location (path, address, | ||
| endpoint) to store your data. It can represent a cloud storage service, an SSH | ||
| server, network-attached storage, or even a directory in the local file system. | ||
| (See all the supported remote storage types in the examples below.) If `url` is | ||
| a relative path, it will be resolved against the current working directory, but | ||
| saved **relative to the config file location** (see LOCAL example below). | ||
| Whenever possible, DVC will create a remote directory if it doesn't exists yet. | ||
| (It won't create an S3 bucket though, and will rely on default access settings.) | ||
|
|
||
| > If you installed DVC via `pip` and plan to use cloud services as remote | ||
| > storage, you might need to install these optional dependencies: `[s3]`, | ||
| > `[azure]`, `[gdrive]`, `[gs]`, `[oss]`, `[ssh]`. Alternatively, use `[all]` to | ||
| > include them all. The command should look like this: `pip install "dvc[s3]"`. | ||
| > (This example installs `boto3` library along with DVC to support S3 storage.) | ||
| This command creates a section in the <abbr>DVC project</abbr>'s | ||
| [config file](/doc/command-reference/config) and optionally assigns a default | ||
|
|
@@ -89,46 +89,6 @@ These are the possible remote storage (protocols) DVC can work with: | |
|
|
||
| <details> | ||
|
|
||
| ### Click for local remote | ||
|
|
||
| A "local remote" is a directory in the machine's file system. | ||
|
|
||
| > While the term may seem contradictory, it doesn't have to be. The "local" part | ||
| > refers to the machine where the project is stored, so it can be any directory | ||
| > accessible to the same system. The "remote" part refers specifically to the | ||
| > project/repository itself. | ||
| Using an absolute path (recommended): | ||
|
|
||
| ```dvc | ||
| $ dvc remote add myremote /tmp/my-dvc-storage | ||
| $ cat .dvc/config | ||
| ... | ||
| ['remote "myremote"'] | ||
| url = /tmp/my-dvc-storage | ||
| ... | ||
| ``` | ||
|
|
||
| > Note that the absolute path `/tmp/my-dvc-storage` is saved as is. | ||
| Using a relative path: | ||
|
|
||
| ```dvc | ||
| $ dvc remote add myremote ../my-dvc-storage | ||
| $ cat .dvc/config | ||
| ... | ||
| ['remote "myremote"'] | ||
| url = ../../my-dvc-storage | ||
| ... | ||
| ``` | ||
|
|
||
| > Note that `../my-dvc-storage` has been resolved relative to the `.dvc/` dir, | ||
| > resulting in `../../my-dvc-storage`. | ||
| </details> | ||
|
|
||
| <details> | ||
|
|
||
| ### Click for Amazon S3 | ||
|
|
||
| > **Note!** Before adding a new remote be sure to login into AWS services and | ||
|
|
@@ -196,7 +156,7 @@ For more information about the variables DVC supports, please visit | |
|
|
||
| <details> | ||
|
|
||
| ### Click for Azure | ||
| ### Click for Microsoft Azure Blob Storage | ||
|
|
||
| ```dvc | ||
| $ dvc remote add myremote azure://my-container-name/path | ||
|
|
@@ -282,15 +242,70 @@ $ dvc remote add myremote gs://bucket/path | |
|
|
||
| <details> | ||
|
|
||
| ### Click for Aliyun OSS | ||
|
|
||
| First you need to setup OSS storage on Aliyun Cloud and then use an S3 style URL | ||
| for OSS storage and make the endpoint value configurable. An example is shown | ||
| below: | ||
|
|
||
| ```dvc | ||
| $ dvc remote add myremote oss://my-bucket/path | ||
| ``` | ||
|
|
||
| To set key id, key secret and endpoint you need to use `dvc remote modify`. | ||
| Example usage is show below. Make sure to use the `--local` option to avoid | ||
| committing your secrets into Git: | ||
|
|
||
| ```dvc | ||
| $ dvc remote modify myremote --local oss_key_id my-key-id | ||
| $ dvc remote modify myremote --local oss_key_secret my-key-secret | ||
| $ dvc remote modify myremote oss_endpoint endpoint | ||
| ``` | ||
|
|
||
| You can also set environment variables and use them later, to set environment | ||
| variables use following environment variables: | ||
|
|
||
| ```dvc | ||
| $ export OSS_ACCESS_KEY_ID="my-key-id" | ||
| $ export OSS_ACCESS_KEY_SECRET="my-key-secret" | ||
| $ export OSS_ENDPOINT="endpoint" | ||
| ``` | ||
|
|
||
| #### Test your OSS storage using docker | ||
|
|
||
| Start a container running an OSS emulator. | ||
|
|
||
| ```dvc | ||
| $ git clone https://github.com/nanaya-tachibana/oss-emulator.git | ||
| $ docker image build -t oss:1.0 oss-emulator | ||
| $ docker run --detach -p 8880:8880 --name oss-emulator oss:1.0 | ||
| ``` | ||
|
|
||
| Setup environment variables. | ||
|
|
||
| ```dvc | ||
| $ export OSS_BUCKET='my-bucket' | ||
| $ export OSS_ENDPOINT='localhost:8880' | ||
| $ export OSS_ACCESS_KEY_ID='AccessKeyID' | ||
| $ export OSS_ACCESS_KEY_SECRET='AccessKeySecret' | ||
| ``` | ||
|
|
||
| > Uses default key id and key secret when they are not given, which gives read | ||
| > access to public read bucket and public bucket. | ||
| </details> | ||
|
|
||
| <details> | ||
|
|
||
| ### Click for SSH | ||
|
|
||
| ```dvc | ||
| $ dvc remote add myremote ssh://[email protected]/path/to/dir | ||
| ``` | ||
|
|
||
| > **Note!** DVC requires both SSH and SFTP access to work with SSH remote | ||
| > storage. Please check that you are able to connect to the remote location with | ||
| > tools like `ssh` and `sftp` (GNU/Linux). | ||
| > storage. Please check that you are able to connect both ways to the remote | ||
| > location, with tools like `ssh` and `sftp` (GNU/Linux). | ||
| <!-- Separate MD quote: --> | ||
|
|
||
|
|
@@ -336,56 +351,41 @@ $ dvc remote add myremote https://example.com/path/to/dir | |
|
|
||
| <details> | ||
|
|
||
| ### Click for Aliyun OSS | ||
|
|
||
| First you need to setup OSS storage on Aliyun Cloud and then use an S3 style URL | ||
| for OSS storage and make the endpoint value configurable. An example is shown | ||
| below: | ||
|
|
||
| ```dvc | ||
| $ dvc remote add myremote oss://my-bucket/path | ||
| ``` | ||
| ### Click for local remote | ||
|
|
||
| To set key id, key secret and endpoint you need to use `dvc remote modify`. | ||
| Example usage is show below. Make sure to use the `--local` option to avoid | ||
| committing your secrets into Git: | ||
| A "local remote" is a directory in the machine's file system. | ||
|
|
||
| ```dvc | ||
| $ dvc remote modify myremote --local oss_key_id my-key-id | ||
| $ dvc remote modify myremote --local oss_key_secret my-key-secret | ||
| $ dvc remote modify myremote oss_endpoint endpoint | ||
| ``` | ||
| > While the term may seem contradictory, it doesn't have to be. The "local" part | ||
| > refers to the machine where the project is stored, so it can be any directory | ||
| > accessible to the same system. The "remote" part refers specifically to the | ||
| > project/repository itself. | ||
| You can also set environment variables and use them later, to set environment | ||
| variables use following environment variables: | ||
| Using an absolute path (recommended): | ||
|
|
||
| ```dvc | ||
| $ export OSS_ACCESS_KEY_ID="my-key-id" | ||
| $ export OSS_ACCESS_KEY_SECRET="my-key-secret" | ||
| $ export OSS_ENDPOINT="endpoint" | ||
| $ dvc remote add myremote /tmp/my-dvc-storage | ||
| $ cat .dvc/config | ||
| ... | ||
| ['remote "myremote"'] | ||
| url = /tmp/my-dvc-storage | ||
| ... | ||
| ``` | ||
|
|
||
| #### Test your OSS storage using docker | ||
|
|
||
| Start a container running an OSS emulator. | ||
|
|
||
| ```dvc | ||
| $ git clone https://github.com/nanaya-tachibana/oss-emulator.git | ||
| $ docker image build -t oss:1.0 oss-emulator | ||
| $ docker run --detach -p 8880:8880 --name oss-emulator oss:1.0 | ||
| ``` | ||
| > Note that the absolute path `/tmp/my-dvc-storage` is saved as is. | ||
| Setup environment variables. | ||
| Using a relative path: | ||
|
|
||
| ```dvc | ||
| $ export OSS_BUCKET='my-bucket' | ||
| $ export OSS_ENDPOINT='localhost:8880' | ||
| $ export OSS_ACCESS_KEY_ID='AccessKeyID' | ||
| $ export OSS_ACCESS_KEY_SECRET='AccessKeySecret' | ||
| $ dvc remote add myremote ../my-dvc-storage | ||
| $ cat .dvc/config | ||
| ... | ||
| ['remote "myremote"'] | ||
| url = ../../my-dvc-storage | ||
| ... | ||
| ``` | ||
|
|
||
| > Uses default key id and key secret when they are not given, which gives read | ||
| > access to public read bucket and public bucket. | ||
| > Note that `../my-dvc-storage` has been resolved relative to the `.dvc/` dir, | ||
| > resulting in `../../my-dvc-storage`. | ||
| </details> | ||
|
|
||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.