Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Leave Open] Fix metadata sorting issue #13753

Closed
wants to merge 6 commits into from
Closed

[Leave Open] Fix metadata sorting issue #13753

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
40162ee
added sorting fix
tasherif-msft Sep 11, 2020
d122f4c
added test
tasherif-msft Sep 11, 2020
2cc2afe
manually set the env var
tasherif-msft Sep 13, 2020
c9c822a
Merge branch 'master' into sorting-issue
tasherif-msft Sep 13, 2020
8db56e0
changed setting
tasherif-msft Sep 13, 2020
e3e643d
removed hardcoded env setting
tasherif-msft Sep 14, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 12 additions & 2 deletions sdk/storage/azure-storage-blob/azure/storage/blob/_shared/authentication.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@

import logging
import sys
import locale

try:
from urllib.parse import urlparse, unquote
Expand All @@ -30,7 +31,7 @@


logger = logging.getLogger(__name__)

locale.setlocale(locale.LC_COLLATE, "")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this a global setting ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it is. I've been looking into ways to create some form of a locale object to sort based on without having to set globally. So far, my findings have been: I can either run a child process and set the locale in it or set the locale right before my sort then undo it right after my sort (by resetting to the original locale).

Copy link
Contributor

@kasobol-msft kasobol-msft Sep 14, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should explore other ways to get sorting right. We shouldn't be setting globals like this as it may interfere with customers application.
I'm not sure if setting and unsetting is good option either. That might lead to race conditions if there's some parallel work being done meanwhile.



# wraps a given exception with the desired exception type
Expand Down Expand Up @@ -92,10 +93,19 @@ def _get_canonicalized_resource(self, request):
def _get_canonicalized_headers(request):
string_to_sign = ''
x_ms_headers = []
x_ms_headers_dict = {}
x_ms_headers_strings = []
for name, value in request.http_request.headers.items():
if name.startswith('x-ms-'):
x_ms_headers.append((name.lower(), value))
x_ms_headers.sort()
for tup in x_ms_headers:
tup_string = ''.join(tup)
x_ms_headers_dict[tup_string] = tup
x_ms_headers_strings.append(tup_string)

x_ms_headers_strings.sort(key=locale.strxfrm)
x_ms_headers = [x_ms_headers_dict[sorted_string] for sorted_string in x_ms_headers_strings]

for name, value in x_ms_headers:
if value is not None:
string_to_sign += ''.join([name, ':', value, '\n'])
Expand Down
146 changes: 146 additions & 0 deletions ...ests/recordings/test_blob_access_conditions.test_header_metadata_sort_in_upload_blob.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,146 @@
interactions:
- request:
body: null
headers:
Accept:
- '*/*'
Accept-Encoding:
- gzip, deflate
Connection:
- keep-alive
Content-Length:
- '0'
User-Agent:
- azsdk-python-storage-blob/12.5.0 Python/3.8.5 (Windows-10-10.0.18362-SP0)
x-ms-date:
- Sun, 13 Sep 2020 18:17:12 GMT
x-ms-version:
- '2019-12-12'
method: PUT
uri: https://storagename.blob.core.windows.net/utcontainerbd791bbe?restype=container
response:
body:
string: ''
headers:
content-length:
- '0'
date:
- Sun, 13 Sep 2020 18:17:12 GMT
etag:
- '"0x8D858113CD5E782"'
last-modified:
- Sun, 13 Sep 2020 18:17:13 GMT
server:
- Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
x-ms-version:
- '2019-12-12'
status:
code: 201
message: Created
- request:
body: hello world
headers:
Accept:
- '*/*'
Accept-Encoding:
- gzip, deflate
Connection:
- keep-alive
Content-Length:
- '11'
Content-Type:
- application/octet-stream
If-None-Match:
- '*'
User-Agent:
- azsdk-python-storage-blob/12.5.0 Python/3.8.5 (Windows-10-10.0.18362-SP0)
x-ms-blob-type:
- BlockBlob
x-ms-date:
- Sun, 13 Sep 2020 18:17:13 GMT
x-ms-version:
- '2019-12-12'
method: PUT
uri: https://storagename.blob.core.windows.net/utcontainerbd791bbe/blob1
response:
body:
string: ''
headers:
content-length:
- '0'
content-md5:
- XrY7u+Ae7tCTyyK7j1rNww==
date:
- Sun, 13 Sep 2020 18:17:12 GMT
etag:
- '"0x8D858113CE74816"'
last-modified:
- Sun, 13 Sep 2020 18:17:13 GMT
server:
- Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
x-ms-content-crc64:
- vo7q9sPVKY0=
x-ms-request-server-encrypted:
- 'true'
x-ms-version:
- '2019-12-12'
status:
code: 201
message: Created
- request:
body: hello world
headers:
Accept:
- '*/*'
Accept-Encoding:
- gzip, deflate
Connection:
- keep-alive
Content-Length:
- '11'
Content-Type:
- application/octet-stream
If-Modified-Since:
- Sun, 13 Sep 2020 18:02:13 GMT
User-Agent:
- azsdk-python-storage-blob/12.5.0 Python/3.8.5 (Windows-10-10.0.18362-SP0)
x-ms-blob-type:
- BlockBlob
x-ms-date:
- Sun, 13 Sep 2020 18:17:13 GMT
x-ms-meta-_a_:
- d
x-ms-meta-i0:
- a
x-ms-meta-i_:
- a
x-ms-version:
- '2019-12-12'
method: PUT
uri: https://storagename.blob.core.windows.net/utcontainerbd791bbe/blob1
response:
body:
string: ''
headers:
content-length:
- '0'
content-md5:
- XrY7u+Ae7tCTyyK7j1rNww==
date:
- Sun, 13 Sep 2020 18:17:12 GMT
etag:
- '"0x8D858113CFBE550"'
last-modified:
- Sun, 13 Sep 2020 18:17:13 GMT
server:
- Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
x-ms-content-crc64:
- vo7q9sPVKY0=
x-ms-request-server-encrypted:
- 'true'
x-ms-version:
- '2019-12-12'
status:
code: 201
message: Created
version: 1
16 changes: 16 additions & 0 deletions sdk/storage/azure-storage-blob/tests/test_blob_access_conditions.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -285,6 +285,22 @@ def test_delete_container_with_if_unmodified_fail(self, resource_group, location

self.assertEqual(StorageErrorCode.condition_not_met, e.exception.error_code)

@GlobalStorageAccountPreparer()
def test_header_metadata_sort_in_upload_blob(self, resource_group, location, storage_account, storage_account_key):
bsc = BlobServiceClient(self.account_url(storage_account, "blob"), storage_account_key, connection_data_block_size=4 * 1024)
self._setup()
data = b'hello world'
container, blob = self._create_container_and_block_blob(
self.container_name, 'blob1', data, bsc)
test_datetime = (datetime.utcnow() -
timedelta(minutes=15))
metadata = {'i0': 'a',
'i_': 'a',
'_a_': 'd'
}
# Act
resp = blob.upload_blob(data, length=len(data), if_modified_since=test_datetime, metadata=metadata)

@GlobalStorageAccountPreparer()
def test_put_blob_with_if_modified(self, resource_group, location, storage_account, storage_account_key):
bsc = BlobServiceClient(self.account_url(storage_account, "blob"), storage_account_key, connection_data_block_size=4 * 1024)
Expand Down