From dc79a105037d087828f58f9bc60cb9c59e80e200 Mon Sep 17 00:00:00 2001 From: meierale Date: Wed, 14 Jul 2021 19:09:18 +0200 Subject: [PATCH 1/9] Authentication examples for Azure Blob Storage Added some examples how to remoge modify dvc for various azure blob storage authentication scenarios. --- .../docs/command-reference/remote/modify.md | 41 +++++++++++++++++-- 1 file changed, 38 insertions(+), 3 deletions(-) diff --git a/content/docs/command-reference/remote/modify.md b/content/docs/command-reference/remote/modify.md index ac1ccc58d1..be679b9c13 100644 --- a/content/docs/command-reference/remote/modify.md +++ b/content/docs/command-reference/remote/modify.md @@ -334,7 +334,7 @@ storage. Whether they're effective depends on each storage platform. - `account_name` (required) - storage account name ```dvc - $ dvc remote modify myremote account_name 'myuser' + $ dvc remote modify myremote account_name 'your_storage_account_name' ``` By default, DVC authenticates using an `account_name` and its @@ -343,7 +343,7 @@ By default, DVC authenticates using an `account_name` and its application. To use a custom authentication method, use the following parameters (listed in order of precedence): -1. `connection_string` is used for authentication if given (all others params +1. `connection_string` is used for authentication if given (all others parameters are ignored). 2. If `tenant_id` and `client_id` or `client_secret` are given, Active Directory (AD) @@ -406,7 +406,42 @@ application. To use a custom authentication method, use the following parameters ```dvc $ dvc remote modify myremote allow_anonymous_login true ``` +**Authentication examples:** +Authentication example with `connection_string`: + +```dvc +$ dvc remote add -d myremote azure://mycontainer/object +$ dvc remote modify --local myremote connection_string 'storageaccountaccesskeyconnectionstring' +$ dvc remote push +``` + +Authentication example with currently logged in az cli user/identity: + +```dvc +$ dvc remote add -d myremote azure://mycontainer/object +$ dvc remote modify myremote account_name 'mystorageaccountname' +$ dvc remote push +``` + +Authentication example with `account_key`: + +```dvc +$ dvc remote add -d myremote azure://mycontainer/object +$ dvc remote modify --local myremote account_name 'mystorageaccountname' +$ dvc remote modify --local myremote account_key 'mystorageaccountaccesskey' +$ dvc remote push +``` + +Authentication example with `sas_token`: + +```dvc +$ dvc remote add -d myremote azure://mycontainer/object +$ dvc remote modify --local myremote account_name 'mystorageaccountname' +$ dvc remote modify --local myremote sas_token 'mysastoken' +$ dvc remote push +``` + Note that Azure remotes can also authenticate via environment variables (instead of `dvc remote modify`). These are tried if none of the params above are set. @@ -419,7 +454,7 @@ $ export AZURE_STORAGE_CONNECTION_STRING='mysecret' For account name and key/token auth: ```dvc -$ export AZURE_STORAGE_ACCOUNT='myuser' +$ export AZURE_STORAGE_ACCOUNT='mystorageaccount' # and $ export AZURE_STORAGE_KEY='mysecret' # or From eb3c5ce0d4e217f4e9369ecf361c49e88dab4098 Mon Sep 17 00:00:00 2001 From: meierale Date: Thu, 15 Jul 2021 07:57:04 +0200 Subject: [PATCH 2/9] Update modify.md minor adjustments due to feedback --- content/docs/command-reference/remote/modify.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/content/docs/command-reference/remote/modify.md b/content/docs/command-reference/remote/modify.md index be679b9c13..464af9672a 100644 --- a/content/docs/command-reference/remote/modify.md +++ b/content/docs/command-reference/remote/modify.md @@ -334,7 +334,7 @@ storage. Whether they're effective depends on each storage platform. - `account_name` (required) - storage account name ```dvc - $ dvc remote modify myremote account_name 'your_storage_account_name' + $ dvc remote modify myremote account_name 'mystorageaccountname' ``` By default, DVC authenticates using an `account_name` and its @@ -343,8 +343,8 @@ By default, DVC authenticates using an `account_name` and its application. To use a custom authentication method, use the following parameters (listed in order of precedence): -1. `connection_string` is used for authentication if given (all others parameters - are ignored). +1. `connection_string` is used for authentication if given (all others params + are ignored / should be removed from the config). 2. If `tenant_id` and `client_id` or `client_secret` are given, Active Directory (AD) [service principal](https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal) From aa00fbfbc0d147702cc3c3905c5f05ade9d9bd65 Mon Sep 17 00:00:00 2001 From: Jorge Orpinel Date: Mon, 19 Jul 2021 02:38:09 -0500 Subject: [PATCH 3/9] ref: mystorageaccountname -> myaccount in `remote modify` --- content/docs/command-reference/remote/modify.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/docs/command-reference/remote/modify.md b/content/docs/command-reference/remote/modify.md index 464af9672a..257a165d4d 100644 --- a/content/docs/command-reference/remote/modify.md +++ b/content/docs/command-reference/remote/modify.md @@ -334,7 +334,7 @@ storage. Whether they're effective depends on each storage platform. - `account_name` (required) - storage account name ```dvc - $ dvc remote modify myremote account_name 'mystorageaccountname' + $ dvc remote modify myremote account_name 'myaccount' ``` By default, DVC authenticates using an `account_name` and its @@ -454,7 +454,7 @@ $ export AZURE_STORAGE_CONNECTION_STRING='mysecret' For account name and key/token auth: ```dvc -$ export AZURE_STORAGE_ACCOUNT='mystorageaccount' +$ export AZURE_STORAGE_ACCOUNT='myaccount' # and $ export AZURE_STORAGE_KEY='mysecret' # or From 8f1c355a09fe581b68ce7003f343d88b72b51a69 Mon Sep 17 00:00:00 2001 From: Jorge Orpinel Date: Mon, 19 Jul 2021 02:52:22 -0500 Subject: [PATCH 4/9] ref: std value samples in remote modify --- content/docs/command-reference/remote/modify.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/content/docs/command-reference/remote/modify.md b/content/docs/command-reference/remote/modify.md index 257a165d4d..f19d4cba06 100644 --- a/content/docs/command-reference/remote/modify.md +++ b/content/docs/command-reference/remote/modify.md @@ -412,7 +412,7 @@ Authentication example with `connection_string`: ```dvc $ dvc remote add -d myremote azure://mycontainer/object -$ dvc remote modify --local myremote connection_string 'storageaccountaccesskeyconnectionstring' +$ dvc remote modify --local myremote connection_string 'mysecret' $ dvc remote push ``` @@ -420,7 +420,7 @@ Authentication example with currently logged in az cli user/identity: ```dvc $ dvc remote add -d myremote azure://mycontainer/object -$ dvc remote modify myremote account_name 'mystorageaccountname' +$ dvc remote modify myremote account_name 'myaccount' $ dvc remote push ``` @@ -428,8 +428,8 @@ Authentication example with `account_key`: ```dvc $ dvc remote add -d myremote azure://mycontainer/object -$ dvc remote modify --local myremote account_name 'mystorageaccountname' -$ dvc remote modify --local myremote account_key 'mystorageaccountaccesskey' +$ dvc remote modify --local myremote account_name 'myaccount' +$ dvc remote modify --local myremote account_key 'mysecret' $ dvc remote push ``` @@ -437,8 +437,8 @@ Authentication example with `sas_token`: ```dvc $ dvc remote add -d myremote azure://mycontainer/object -$ dvc remote modify --local myremote account_name 'mystorageaccountname' -$ dvc remote modify --local myremote sas_token 'mysastoken' +$ dvc remote modify --local myremote account_name 'myaccount' +$ dvc remote modify --local myremote sas_token 'mysecret' $ dvc remote push ``` From 2655da0620487cb1fe4e4f5968fa0848da6ed322 Mon Sep 17 00:00:00 2001 From: meierale Date: Fri, 23 Jul 2021 12:53:49 +0200 Subject: [PATCH 5/9] Update modify.md change L347 back to how it originally was. --- content/docs/command-reference/remote/modify.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/docs/command-reference/remote/modify.md b/content/docs/command-reference/remote/modify.md index f19d4cba06..8022ce679f 100644 --- a/content/docs/command-reference/remote/modify.md +++ b/content/docs/command-reference/remote/modify.md @@ -344,7 +344,7 @@ application. To use a custom authentication method, use the following parameters (listed in order of precedence): 1. `connection_string` is used for authentication if given (all others params - are ignored / should be removed from the config). + are ignored). 2. If `tenant_id` and `client_id` or `client_secret` are given, Active Directory (AD) [service principal](https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal) From df5c51556d237cb62edc649ada661049eceaf2c6 Mon Sep 17 00:00:00 2001 From: meierale Date: Fri, 23 Jul 2021 13:02:00 +0200 Subject: [PATCH 6/9] Update modify.md make `az cli` credential example the first example, add note about contributor role. --- content/docs/command-reference/remote/modify.md | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/content/docs/command-reference/remote/modify.md b/content/docs/command-reference/remote/modify.md index 8022ce679f..8c406a934c 100644 --- a/content/docs/command-reference/remote/modify.md +++ b/content/docs/command-reference/remote/modify.md @@ -407,20 +407,22 @@ application. To use a custom authentication method, use the following parameters $ dvc remote modify myremote allow_anonymous_login true ``` **Authentication examples:** - -Authentication example with `connection_string`: - + +Authentication example with currently logged in az cli user/identity: + ```dvc $ dvc remote add -d myremote azure://mycontainer/object -$ dvc remote modify --local myremote connection_string 'mysecret' +$ dvc remote modify myremote account_name 'myaccount' $ dvc remote push ``` - -Authentication example with currently logged in az cli user/identity: +Note: the above example requires at least + `Storage Blob Data Contributor` role on `myaccount`. + +Authentication example with `connection_string`: ```dvc $ dvc remote add -d myremote azure://mycontainer/object -$ dvc remote modify myremote account_name 'myaccount' +$ dvc remote modify --local myremote connection_string 'mysecret' $ dvc remote push ``` From f1513021525ddbf12d0570fcd45b4f532b397347 Mon Sep 17 00:00:00 2001 From: Jorge Orpinel Date: Wed, 28 Jul 2021 06:12:38 +0000 Subject: [PATCH 7/9] ref: clarify account_name param in remote modify per https://github.com/iterative/dvc.org/pull/2634#pullrequestreview-709245680 --- .../docs/command-reference/remote/modify.md | 27 ++++++++++--------- 1 file changed, 15 insertions(+), 12 deletions(-) diff --git a/content/docs/command-reference/remote/modify.md b/content/docs/command-reference/remote/modify.md index cd4adb0fab..31de1cd7b9 100644 --- a/content/docs/command-reference/remote/modify.md +++ b/content/docs/command-reference/remote/modify.md @@ -331,7 +331,8 @@ storage. Whether they're effective depends on each storage platform. Note that if the given container name isn't found in your account, DVC will attempt to create it. -- `account_name` (required) - storage account name +- `account_name` - storage account name. Required for every authentication + method except `connection_string` (which already includes it). ```dvc $ dvc remote modify myremote account_name 'myaccount' @@ -371,8 +372,8 @@ $ dvc remote modify --system myremote To use a custom authentication method, use the following parameters (listed in order of precedence): -1. `connection_string` is used for authentication if given (all others params - are ignored). +1. `connection_string` is used for authentication if given (`account_name` is + ignored). 2. If `tenant_id` and `client_id` or `client_secret` are given, Active Directory (AD) [service principal](https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal) @@ -434,17 +435,19 @@ order of precedence): ```dvc $ dvc remote modify myremote allow_anonymous_login true ``` -**Authentication examples:** - + + **Authentication examples:** + Authentication example with currently logged in az cli user/identity: - + ```dvc $ dvc remote add -d myremote azure://mycontainer/object $ dvc remote modify myremote account_name 'myaccount' $ dvc remote push ``` -Note: the above example requires at least - `Storage Blob Data Contributor` role on `myaccount`. + +Note: the above example requires at least `Storage Blob Data Contributor` role +on `myaccount`. Authentication example with `connection_string`: @@ -453,7 +456,7 @@ $ dvc remote add -d myremote azure://mycontainer/object $ dvc remote modify --local myremote connection_string 'mysecret' $ dvc remote push ``` - + Authentication example with `account_key`: ```dvc @@ -462,7 +465,7 @@ $ dvc remote modify --local myremote account_name 'myaccount' $ dvc remote modify --local myremote account_key 'mysecret' $ dvc remote push ``` - + Authentication example with `sas_token`: ```dvc @@ -470,8 +473,8 @@ $ dvc remote add -d myremote azure://mycontainer/object $ dvc remote modify --local myremote account_name 'myaccount' $ dvc remote modify --local myremote sas_token 'mysecret' $ dvc remote push -``` - +``` + Note that Azure remotes can also authenticate via environment variables (instead of `dvc remote modify`). These are tried if none of the params above are set. From 3904abbdc1d05e11a14d2fd7fe38ce67c51bf784 Mon Sep 17 00:00:00 2001 From: Jorge Orpinel Date: Wed, 28 Jul 2021 06:49:07 +0000 Subject: [PATCH 8/9] ref: move Azure examples to the bottom of remote modify --- .../docs/command-reference/remote/modify.md | 102 +++++++++--------- 1 file changed, 53 insertions(+), 49 deletions(-) diff --git a/content/docs/command-reference/remote/modify.md b/content/docs/command-reference/remote/modify.md index 31de1cd7b9..815536c86f 100644 --- a/content/docs/command-reference/remote/modify.md +++ b/content/docs/command-reference/remote/modify.md @@ -374,14 +374,17 @@ order of precedence): 1. `connection_string` is used for authentication if given (`account_name` is ignored). -2. If `tenant_id` and `client_id` or `client_secret` are given, Active Directory - (AD) - [service principal](https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal) - auth is performed. +2. If `tenant_id` and `client_id`, `client_secret` are given, Active Directory + (AD) [service principal] auth is performed. 3. DVC will try next to connect with `account_key` or `sas_token` (in this order) if either are provided. 4. If `allow_anonymous_login` is set to `True`, then DVC will try to connect - [anonymously](https://docs.microsoft.com/en-us/azure/storage/blobs/anonymous-read-access-configure). + [anonymously]. + +[service principal]: + https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal +[anonymously]: + https://docs.microsoft.com/en-us/azure/storage/blobs/anonymous-read-access-configure > The authentication values below may contain sensitive user info. Therefore, > it's safer to use the `--local` flag so they're written to a Git-ignored @@ -392,34 +395,35 @@ order of precedence): (recommended). ```dvc - $ dvc remote modify --local myremote connection_string 'mysecret' + $ dvc remote modify --local + myremote connection_string 'mysecret' ``` * `tenant_id` - tenant ID for AD _service principal_ authentication (requires `client_id` and `client_secret` along with this): ```dvc - $ dvc remote modify --local myremote tenant_id 'directory-id' + $ dvc remote modify --local myremote tenant_id 'mytenant' ``` * `client_id` - client ID for _service principal_ authentication (when `tenant_id` is set): ```dvc - $ dvc remote modify --local myremote client_id 'client-id' + $ dvc remote modify --local myremote client_id 'myclient' ``` * `client_secret` - client Secret for _service principal_ authentication (when `tenant_id` is set): ```dvc - $ dvc remote modify --local myremote client_secret 'client-secret' + $ dvc remote modify --local myremote client_secret 'mysecret' ``` * `account_key` - storage account key: ```dvc - $ dvc remote modify --local myremote account_key 'mysecret' + $ dvc remote modify --local myremote account_key 'mykey' ``` * `sas_token` - shared access signature token: @@ -436,45 +440,6 @@ order of precedence): $ dvc remote modify myremote allow_anonymous_login true ``` - **Authentication examples:** - -Authentication example with currently logged in az cli user/identity: - -```dvc -$ dvc remote add -d myremote azure://mycontainer/object -$ dvc remote modify myremote account_name 'myaccount' -$ dvc remote push -``` - -Note: the above example requires at least `Storage Blob Data Contributor` role -on `myaccount`. - -Authentication example with `connection_string`: - -```dvc -$ dvc remote add -d myremote azure://mycontainer/object -$ dvc remote modify --local myremote connection_string 'mysecret' -$ dvc remote push -``` - -Authentication example with `account_key`: - -```dvc -$ dvc remote add -d myremote azure://mycontainer/object -$ dvc remote modify --local myremote account_name 'myaccount' -$ dvc remote modify --local myremote account_key 'mysecret' -$ dvc remote push -``` - -Authentication example with `sas_token`: - -```dvc -$ dvc remote add -d myremote azure://mycontainer/object -$ dvc remote modify --local myremote account_name 'myaccount' -$ dvc remote modify --local myremote sas_token 'mysecret' -$ dvc remote push -``` - Note that Azure remotes can also authenticate via environment variables (instead of `dvc remote modify`). These are tried if none of the params above are set. @@ -1112,3 +1077,42 @@ profile = myuser [core] remote = myremote ``` + +## Example: Some Azure authentication methods + +Using a default identity (e.g. credentials set by `az cli`): + +```dvc +$ dvc remote add -d myremote azure://mycontainer/object +$ dvc remote modify myremote account_name 'myaccount' +$ dvc remote push +``` + +> Note that this may require the `Storage Blob Data Contributor` and other roles +> on the account. + +Using a `connection_string`: + +```dvc +$ dvc remote add -d myremote azure://mycontainer/object +$ dvc remote modify --local myremote connection_string 'mysecret' +$ dvc remote push +``` + +Using `account_key`: + +```dvc +$ dvc remote add -d myremote azure://mycontainer/object +$ dvc remote modify --local myremote account_name 'myaccount' +$ dvc remote modify --local myremote account_key 'mysecret' +$ dvc remote push +``` + +Using `sas_token`: + +```dvc +$ dvc remote add -d myremote azure://mycontainer/object +$ dvc remote modify --local myremote account_name 'myaccount' +$ dvc remote modify --local myremote sas_token 'mysecret' +$ dvc remote push +``` From c80e76817175812a5a7925f588cdac1e70f957af Mon Sep 17 00:00:00 2001 From: Jorge Orpinel Date: Wed, 28 Jul 2021 15:59:52 +0000 Subject: [PATCH 9/9] ref: link to Azure auth examples from remote add/modify --- content/docs/command-reference/remote/add.md | 15 ++++++++++----- content/docs/command-reference/remote/modify.md | 2 ++ 2 files changed, 12 insertions(+), 5 deletions(-) diff --git a/content/docs/command-reference/remote/add.md b/content/docs/command-reference/remote/add.md index ff283241d6..e4fe375c9b 100644 --- a/content/docs/command-reference/remote/add.md +++ b/content/docs/command-reference/remote/add.md @@ -150,11 +150,16 @@ $ dvc remote add -d myremote azure://mycontainer/path $ dvc remote modify myremote account_name 'myuser' ``` -By default, DVC authenticates using an `account_name` and its -[default credential](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html) -(if any), which uses certain environment variables or a signed-in Microsoft -application. To use a custom authentication method, use the parameters described -in `dvc remote modify`. +By default, DVC authenticates using an `account_name` and its [default +credential] (if any), which uses environment variables (e.g. set by `az cli`) or +a Microsoft application. + +[default credential]: + https://docs.microsoft.com/en-us/python/api/azure-identity/azure.identity.defaultazurecredential + +To use a custom authentication method, use the parameters described in +`dvc remote modify`. See some +[examples](#example-some-azure-authentication-methods). diff --git a/content/docs/command-reference/remote/modify.md b/content/docs/command-reference/remote/modify.md index 815536c86f..81eddf57ef 100644 --- a/content/docs/command-reference/remote/modify.md +++ b/content/docs/command-reference/remote/modify.md @@ -369,6 +369,8 @@ $ dvc remote modify --system myremote +> See some [Azure auth examples](#example-some-azure-authentication-methods). + To use a custom authentication method, use the following parameters (listed in order of precedence):