TLS error when using HTTPS/TLS URL (not permitted for non-TLS protected (non-https) URLs.)

[nofunatall]

Ubuntu 19.10
Linux rio-t460p 5.3.0-26-generic #28-Ubuntu SMP Wed Dec 18 05:37:46 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Python 3.7.5
azure-core==1.2.0
azure-identity==1.2.0
azure-storage-blob==12.1.0

Describe the bug
When trying to do a batch delete blob using a blob client created using a HTTPS URL the code errors with:

"Bearer token authentication is not permitted for non-TLS protected (non-https) URLs."
azure.core.exceptions.ServiceRequestError: Bearer token authentication is not permitted for non-TLS protected (non-https) URLs.

This same blob client connection can successfully be used to list blobs in a container.
The credentials used to create the client are service principal with rbac (client/secret/tenant) and the role is set to "Storage Blob Data Owner".

Steps to reproduce
Here is some example code to reproduce the error:

STORAGE_URL = 'https://' + STORAGE_ACCOUNT + '.blob.core.windows.net'
client_credentials = ClientSecretCredential(client_id=AZURE_CLIENT, client_secret=AZURE_SECRET, tenant_id=AZURE_TENANT)
blob_service_client = BlobServiceClient(account_url=STORAGE_URL, credential=client_credentials)
container_client = blob_service_client.get_container_client(STORAGE_CONTAINER)
def run_azure_blob_list():
  result = []
  blob_list = container_client.list_blobs(name_starts_with=STORAGE_FOLDER)
  for blob in blob_list:
    if blob.last_modified < IGNORE_DATE: 
      result.append([blob.name, blob.last_modified])
  return result
blob_list = run_azure_blob_list()
container_client.delete_blobs(*blob_list)

Container endpoint:
print(container_client.primary_endpoint)

https://REDACTED.blob.core.windows.net/uploads

[annatisch]

Thanks for the report @nofunatall - I will investigate this today :)

[annatisch]

Sorry for the delay @nofunatall!
Adding @xiafu-msft from the storage team for visibility.

[annatisch]

Update:
I have managed to repro this locally and have confirmed that it's actually an incompatibility in the forming the Storage "batch" requests, and the way oauth policy in azure-core enforcing the 'https' URL scheme.
This fix will need to be added to Azure core, so updating the labels.
Adding @lmazuel and @chlowell
The issue is that each part in the multi-part requests for storage do not use a fully qualified URL, instead it's just a path stub, e.g. /my-container/my-blob.txt

Therefore, when we hit the TLS check in the BearerTokenCredentialsPolicy the request is deemed invalid.

Details on the Batch request formatting are here:
https://docs.microsoft.com/en-us/rest/api/storageservices/blob-batch
Note that each "subrequest" is authenticated independently - and each subrequest is fed through the pipeline independently in order to generate those auth headers.

[lmazuel]

I suggest we don't apply this check when the URL is partial (parse and check the scheme, instead of startswith)

[annatisch]

@lmazuel, @xiangyan99 - did we land on a solution for this issue?

[lmazuel]

Yes, waiting for #9821 to make it to azure-core (we missed the 1.3 timeline)

[annatisch]

This fix has been merged into master and will ship with the next release :)

[Petermarcu]

Should we close this issue now?

[annatisch]

Yes :)

[vsalvino]

Is this fix going out soon (1.4.*) - or is it not yet known when this will be released? I am still experiencing it in azure-core==1.4.0 and azure-storage-blobs==12.3.0.

[annatisch]

Thanks @vsalvino - I had forgotten that the blobs update hadn't actually been released yet. Re-opening for now.
@xiafu-msft - do you have an idea as to when the next blobs release will be?

[lmazuel]

Released part of 12.3.1 (see tags of this commit)

[DroidUnknown]

Windows 11 preview
azure-core==1.16.0
azure-identity==1.5.0
azure-mgmt-synapse==2.0.0

I am getting this same error when trying to access Synapse

import os
from azure.identity import ClientSecretCredential
from azure.mgmt.synapse import SynapseManagementClient

clientId = "********-****-****-****-************";
secret = "D0i******************************";
tenant = "********-****-****-****-************";
subId = "********-****-****-****-************";
resourceGroupName = "******-***-***";
workspaceName = "***********";


credentials = ClientSecretCredential(tenant_id=tenant, client_id=clientId, client_secret=secret)

client = SynapseManagementClient(credentials, subId, base_url="dev.azuresynapse.net")

workspace = client.workspaces.get(resourceGroupName, workspaceName)
print(workspace)

This is the exception -> azure.core.exceptions.ServiceRequestError: Bearer token authentication is not permitted for non-TLS protected (non-https) URLs.

[chlowell]

That's by design in this case. Your base_url doesn't include a scheme, and the authentication policy requires that requests are protected by TLS, i.e. that their URLS begin with "https", because client requests carry access tokens. The solution then would be to add the scheme: base_url="https://dev.azuresynapse.net".

All that said, I think this is still an incorrect value. SynapseManagementClient manages Synapse resources via Azure Resource Management (ARM). base_url should be an ARM endpoint like https://management.azure.com, which is the default value and correct for Azure Public Cloud. So, you only need to specify base_url when managing resources in sovereign or private clouds. Is that what you're trying to do?

[DroidUnknown]

No I was simply trying to setup Synapse using python SDK. This is my first time trying to do something on azure and did not really understand the details. https://management.azure.com works in this case. Thanks