Authentication Error when TableClient created with from_table_url

[pavelm10]

• azure-data-tables:
• 12.4.2:
• Ubuntu 22.04 LTS:
• 3.9.14, 3.10.7:

Describe the bug
I am getting azure.core.exceptions.ClientAuthenticationError when querying TableClient that is initialized with TableClient.from_table_url() method when using azure-data-tables = "12.4.2". The error does not occur for azure-data-tables = "12.4.1"

I am not getting the error when the TableClient is initialized with constructor.

The traceback:

Traceback (most recent call last):
  File "/home/marek/temp/datatables/.venv/lib/python3.9/site-packages/azure/data/tables/_models.py", line 360, in _get_next_cb
    return self._command(
  File "/home/marek/temp/datatables/.venv/lib/python3.9/site-packages/azure/core/tracing/decorator.py", line 78, in wrapper_use_tracer
    return func(*args, **kwargs)
  File "/home/marek/temp/datatables/.venv/lib/python3.9/site-packages/azure/data/tables/_generated/operations/_operations.py", line 985, in query_entities
    raise HttpResponseError(response=response, model=error)
azure.core.exceptions.HttpResponseError: Operation returned an invalid status 'Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.'
Content: {"odata.error":{"code":"AuthenticationFailed","message":{"lang":"en-US","value":"Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:8d1cc3c1-c002-00e9-19be-45e901000000\nTime:2023-02-21T06:33:41.8347647Z"}}}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/marek/temp/datatables/datatables/app.py", line 40, in <module>
    df = load_table_from_url()
  File "/home/marek/temp/datatables/datatables/app.py", line 19, in load_table_from_url
    return query_table(table_service)
  File "/home/marek/temp/datatables/datatables/app.py", line 14, in query_table
    return pd.DataFrame.from_records(entities)
  File "/home/marek/temp/datatables/.venv/lib/python3.9/site-packages/pandas/core/frame.py", line 2297, in from_records
    first_row = next(data)
  File "/home/marek/temp/datatables/.venv/lib/python3.9/site-packages/azure/core/paging.py", line 132, in __next__
    return next(self._page_iterator)
  File "/home/marek/temp/datatables/.venv/lib/python3.9/site-packages/azure/core/paging.py", line 76, in __next__
    self._response = self._get_next(self.continuation_token)
  File "/home/marek/temp/datatables/.venv/lib/python3.9/site-packages/azure/data/tables/_models.py", line 371, in _get_next_cb
    _process_table_error(error)
  File "/home/marek/temp/datatables/.venv/lib/python3.9/site-packages/azure/data/tables/_error.py", line 203, in _process_table_error
    _reraise_error(decoded_error)
  File "/home/marek/temp/datatables/.venv/lib/python3.9/site-packages/azure/data/tables/_error.py", line 190, in _reraise_error
    raise decoded_error.with_traceback(exc_traceback)
  File "/home/marek/temp/datatables/.venv/lib/python3.9/site-packages/azure/data/tables/_models.py", line 360, in _get_next_cb
    return self._command(
  File "/home/marek/temp/datatables/.venv/lib/python3.9/site-packages/azure/core/tracing/decorator.py", line 78, in wrapper_use_tracer
    return func(*args, **kwargs)
  File "/home/marek/temp/datatables/.venv/lib/python3.9/site-packages/azure/data/tables/_generated/operations/_operations.py", line 985, in query_entities
    raise HttpResponseError(response=response, model=error)
azure.core.exceptions.ClientAuthenticationError: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:8d1cc3c1-c002-00e9-19be-45e901000000
Time:2023-02-21T06:33:41.8347647Z
ErrorCode:AuthenticationFailed
Content: {"odata.error":{"code":"AuthenticationFailed","message":{"lang":"en-US","value":"Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:8d1cc3c1-c002-00e9-19be-45e901000000\nTime:2023-02-21T06:33:41.8347647Z"}}}

To Reproduce
Using Poetry pyproject.toml:

[tool.poetry]
name = "datatables"
version = "0.1.0"
description = ""
authors = ["Joe Doe <[email protected]>"]

[tool.poetry.dependencies]
python = "^3.9"
azure-core = "^1.26.3"
azure-data-tables = "12.4.2"
azure-identity = "^1.12.0"
pandas = "^1.5.3"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Python code:

import pandas as pd

from azure.identity import AzureCliCredential
from azure.data.tables import TableClient

SAS_URL = 'sas_url_to_dummy_table'
TABLE_NAME = 'DummyTable'


def query_table(table_service: TableClient):
    entities = table_service.query_entities(
            query_filter='PartitionKey eq @pk',
            parameters={'pk': 'dummy-pk'},
        )
    return pd.DataFrame.from_records(entities)


def load_table_from_url():
    table_service = TableClient.from_table_url(SAS_URL)
    return query_table(table_service)


def load_table_with_credential(credential):
    table_service = TableClient(
        endpoint='https://dummy.table.core.windows.net/',
        table_name=TABLE_NAME,
        credential=credential,
    )
    return query_table(table_service)


if __name__ == '__main__':
    credential = AzureCliCredential()

    # This works for both azure-data-tables 12.4.2 and 12.4.1
    df = load_table_with_credential(credential)
    print(df.head())

    # This works ONLY for azure-data-tables 12.4.1
    df = load_table_from_url()
    print(df.head())

Expected behavior
Querying table client constructed in any way does not result in azure.core.exceptions.ClientAuthenticationError when the user has sufficient access rights.

Additional context
Tested only for Python 3.9.14 and 3.10.7.

[annatisch]

Thanks for the report @pavelm10 - we'll investigate asap!
A couple of quick questions - are you using a Storage Tables account or Cosmos Tables account?
How are the SAS URLs being generated?

@YalinLi0312 - could you please investigate the changes made since 12.4.1 to see what might have caused this? I'm wondering if the SAS parameters are being stripped...

[pavelm10]

@annatisch thanks for quick reaction.

• I am using Storage Tables account.
• SAS URL is generated from Azure portal: storage account -> shared access signature

[YalinLi0312]

Hi @pavelm10 , I can't reproduce the error with your sample above in 12.4.2 so far.
I'm not sure how's your SAS_URL get constructed. Can you double check if your SAS_URL is in format https://<TABLES_STORAGE_ACCOUNT_NAME>.table.core.windows.net/<TABLE_NAME><SAS_TOKEN>?

[pavelm10]

Hi @YalinLi0312

I checked again, the SAS_URL is in the format you stated:

https://<storage-account>.table.core.windows.net/<TableName>?sv=2021-06-08&ss=t&srt=sco&sp=rwdlacu&se=2023-02-24T18:24:15Z&st=2023-02-24T10:24:15Z&spr=https&sig=<SIGNATURE>

[xiangyan99]

@YalinLi0312 any updates?

[JayP718]

Getting a similar error, any update?

[JayP718]

Verified seems to be a breaking error in 12.4.2, works fine in prior version

[YalinLi0312]

Hi @pavelm10 and @JayP718 , thanks for your patience!

Based on my investigation, it is a bug to serialize filter in new Serializer in our release 12.4.2. We are working to fix it and will prepare another release soon.
To unblock your work, I will recommend using version 12.4.1 for now. And I'll update here once the new release is ready.

Thanks

[msyyc]

I find the root cause:
(1) Why does 12.4.2 fail?

Both of path_format_arguments and request.url has query params which results in wrong http request url:

(2) Why does 12.4.1 work?

It has different order to format url which could avoid this issue.

[lmazuel]

Looking more deeply into the code, I feel the logic of from_table_url with a SAS token in the url is flawed, and this works before by chance more than by design. The low level code is actually cleaner now, and expose the flaw more explicitly. This is a funny example of bug compensating another bug, and when you fix one you discover the second one.

This would require a deeper fix in the table code. The cleanest workaround right now while using the latest SDK, would be to split the url that contains a SAS Token, and call the SDK with an explicit AzureSasCredential containing that token:
https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/tables/azure-data-tables#creating-the-client-from-a-sas-token

The complete fix will require some redesign, and may take a few weeks to happen.

[github-actions]

Hi @pavelm10. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text "/unresolve" to remove the "issue-addressed" label and continue the conversation.

[YalinLi0312]

Hi @pavelm10 , the bug has been fixed in azure-core, you can try with the latest version of azure-core: https://pypi.org/project/azure-core/1.27.1/

Please let us know if there's any further issue.

Thanks

[github-actions]

Hi @pavelm10, since you haven’t asked that we /unresolve the issue, we’ll close this out. If you believe further discussion is needed, please add a comment /unresolve to reopen the issue.