Signature Error for Query Containing Spaces Encoded as +

[tustvold]

Which service(blob, file, queue, table) does this issue concern?

Blob, but potentially others

Which version of the Azurite was used?

3.18.0

Where do you get Azurite? (npm, DockerHub, NuGet, Visual Studio Code Extension)

mcr.microsoft.com/azure-storage/azurite

What's the Node.js version?

What problem was encountered?

As described in the URL standard spaces in a query string may be encoded as +, and not %20. This is what the standard Rust URL library will always do. As per the specification for Azure this will be URL-decoded as part of generating the canonical string to sign.

For example, GET http://localhost:10000/devstoreaccount1/tustvold?restype=container&comp=list&prefix=foo+bar%2F would result in

GET





Wed, 28 Sep 2022 16:38:41 GMT





x-ms-version:2021-08-06
/devstoreaccount1/devstoreaccount1/tustvold
comp:list
prefix:foo bar/
restype:container

However, making this request to azurite returns a signature error. My guess is it is only percent decoding the query parameters and not url decoding them, and therefore ends up with a different string to sign.

The same request to a Azure proper succeeds

Steps to reproduce the issue?

Noticed in apache/arrow-rs#2801, let me know if you need a more specific example

Have you found a mitigation/solution?

No

[blueww]

@tustvold

Thanks for raising this issue!

We will investigate it, but might will need time.

Before we fix it, would you like to use "%20" instead of "+" to workaround this issue?

[tustvold]

Thank you for the quick response, unfortunately the Rust URL library doesn't provide a way to configure its escaping, at least as far as I'm aware.

We only use azurite for tests, and so I have just suppressed the test for this behaviour, and so there is no major rush from my end.

[blueww]

@tustvold

It looks this issue not repro on my machine, I tried Azurite 3.20.1 + Node 16.15.1, and Azurite 3.10.0 + Node 12.13.0 (OS is Windows 10 Enterprise 22H2), both work well in list blob from container with "prefix=foo+bar%2F" in the URI query.
See following Azurite debug log, the prefix is recognized as "foo bar/" in Azurite.

Would you please share more details about you test environment, like the OS, detail node version...?
And could you upgrade Azurite to latest and node to latest, and see if the issue still repro?

2022-11-03T03:14:03.153Z 53cee751-6c9c-43ac-9044-7a155e09b27e info: BlobStorageContextMiddleware: RequestMethod=GET RequestURL=http://127.0.0.1/devstoreaccount1/sparkconnectortests1?restype=container&comp=list&prefix=foo+bar%2F RequestHeaders:{"user-agent":"Mozilla/5.0 (Windows NT; Windows NT 10.0; zh-CN) WindowsPowerShell/5.1.19041.1682","host":"127.0.0.1:10000"} ClientIP=127.0.0.1 Protocol=http HTTPVersion=1.1
2022-11-03T03:14:03.154Z 53cee751-6c9c-43ac-9044-7a155e09b27e info: BlobStorageContextMiddleware: Account=devstoreaccount1 Container=sparkconnectortests1 Blob=
2022-11-03T03:14:03.154Z 53cee751-6c9c-43ac-9044-7a155e09b27e verbose: DispatchMiddleware: Dispatching request...
2022-11-03T03:14:03.155Z 53cee751-6c9c-43ac-9044-7a155e09b27e info: DispatchMiddleware: Operation=Container_ListBlobFlatSegment
2022-11-03T03:14:03.156Z 53cee751-6c9c-43ac-9044-7a155e09b27e verbose: AuthenticationMiddlewareFactory:createAuthenticationMiddleware() Validating authentications.
2022-11-03T03:14:03.156Z 53cee751-6c9c-43ac-9044-7a155e09b27e info: PublicAccessAuthenticator:validate() Start validation against public access.
2022-11-03T03:14:03.156Z 53cee751-6c9c-43ac-9044-7a155e09b27e debug: PublicAccessAuthenticator:validate() Getting account properties...
2022-11-03T03:14:03.156Z 53cee751-6c9c-43ac-9044-7a155e09b27e debug: PublicAccessAuthenticator:validate() Retrieved account name from context: devstoreaccount1, container: sparkconnectortests1, blob: 
2022-11-03T03:14:03.157Z 53cee751-6c9c-43ac-9044-7a155e09b27e debug: PublicAccessAuthenticator:validate() Public access type for container sparkconnectortests1 is "container"
2022-11-03T03:14:03.157Z 53cee751-6c9c-43ac-9044-7a155e09b27e debug: PublicAccessAuthenticator:validate() Operation Container_ListBlobFlatSegment is in container level public access list. Validation passed.
2022-11-03T03:14:03.157Z 53cee751-6c9c-43ac-9044-7a155e09b27e verbose: DeserializerMiddleware: Start deserializing...
2022-11-03T03:14:03.158Z 53cee751-6c9c-43ac-9044-7a155e09b27e info: HandlerMiddleware: DeserializedParameters={"options":{"prefix":"foo bar/","include":[]},"restype":"container","comp":"list"}
2022-11-03T03:14:03.159Z 53cee751-6c9c-43ac-9044-7a155e09b27e verbose: SerializerMiddleware: Start serializing...
2022-11-03T03:14:03.165Z 53cee751-6c9c-43ac-9044-7a155e09b27e debug: Serializer: Raw response body string is <?xml version="1.0" encoding="UTF-8" standalone="yes"?><EnumerationResults ServiceEndpoint="http://127.0.0.1:10000/devstoreaccount1" ContainerName="sparkconnectortests1"><Prefix>foo bar/</Prefix><Marker/><MaxResults>5000</MaxResults><Delimiter/><Blobs><Blob><Name>foo bar/12</Name><Snapshot/><Properties><Creation-Time>Thu, 03 Nov 2022 03:01:46 GMT</Creation-Time><Last-Modified>Thu, 03 Nov 2022 03:02:12 GMT</Last-Modified><Etag>0x222EF5496A0D8E0</Etag><Content-Length>10485760</Content-Length><Content-Type>application/octet-stream</Content-Type><Content-MD5>2MQVfW/7hIcoV3f3wreX9A==</Content-MD5><BlobType>BlockBlob</BlobType><LeaseStatus>unlocked</LeaseStatus><LeaseState>available</LeaseState><AccessTier>Hot</AccessTier><AccessTierInferred>true</AccessTierInferred></Properties></Blob><Blob><Name>foo bar/56</Name><Snapshot/><Properties><Creation-Time>Thu, 03 Nov 2022 03:02:57 GMT</Creation-Time><Last-Modified>Thu, 03 Nov 2022 03:02:57 GMT</Last-Modified><Etag>0x24B0A4184BBB640</Etag><Content-Length>10485760</Content-Length><Content-Type>application/octet-stream</Content-Type><Content-MD5>2MQVfW/7hIcoV3f3wreX9A==</Content-MD5><BlobType>BlockBlob</BlobType><LeaseStatus>unlocked</LeaseStatus><LeaseState>available</LeaseState><AccessTier>Hot</AccessTier><AccessTierInferred>true</AccessTierInferred></Properties></Blob></Blobs><NextMarker/></EnumerationResults>
2022-11-03T03:14:03.165Z 53cee751-6c9c-43ac-9044-7a155e09b27e info: Serializer: Start returning stream body.
2022-11-03T03:14:03.165Z 53cee751-6c9c-43ac-9044-7a155e09b27e info: EndMiddleware: End response. TotalTimeInMS=12 StatusCode=200 StatusMessage=OK Headers={"server":"Azurite-Blob/3.10.0","content-type":"application/xml","x-ms-request-id":"53cee751-6c9c-43ac-9044-7a155e09b27e","x-ms-version":"2020-04-08","date":"Thu, 03 Nov 2022 03:14:03 GMT"}

[tustvold]

I have double checked this occurs with the latest version. How are you making the request to azurite, are you including an authentication payload, as this is where the issue is?

The corresponding curl command is

$ curl "http://localhost:10000/devstoreaccount1/test-bucket?restype=container&comp=list&prefix=foo+bar%2F" -H 'x-ms-version: 2021-08-06' -H 'authorization: SharedKey devstoreaccount1:fxoNOIvWQyc7R3pHkKRt5ye5Hi7TZHHBWKPZfSav/P0=' -H 'date: Sat, 05 Nov 2022 21:44:31 GMT'
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Error>
  <Code>AuthorizationFailure</Code>
  <Message>Server failed to authenticate the request. Make sure the value of the Authorization header is formed correctly including the signature.
RequestId:a0fe2653-e41a-4258-bf15-b41bd905b8fd
Time:2022-11-05T21:46:59.305Z</Message>
</Error>⏎                                            

[blueww]

@tustvold
Thanks for the info!
I can repro it with shared key credential now.

It looks the problem is caused by decodeURIComponent() used in following line can't pare "+" to space.

Azurite/src/blob/authentication/BlobSharedKeyAuthenticator.ts

Line 311 in 2453337

canonicalizedResourceString += `\n${key}:${decodeURIComponent(

decodeURIComponent("a%20b")
'a b'
decodeURIComponent("a+b")
'a+b'

I will look into it and update you later.

[blueww]

The fix is already released in https://github.com/Azure/Azurite/releases/tag/v3.21.0