Encode asterisks in the canonical request path

[davidvgalbraith]

Hey! Please consider the following:

var https = require('https');
var aws4 = require('aws4');

var opts = { 
    service: 'es',
    region: process.env.REGION,
    headers: { 'Content-Type': 'application/json' },
    host: process.env.HOST,
    path: '/*/*/_search/',
    body: '{"query":{"match_all":{}}}' 
}

https.request(aws4.sign(opts), function(res) {
    res.pipe(process.stdout);
}).end(opts.body);

When I spin up an Amazon Elasticsearch Service instance and set process.env.REGION to the appropriate region and set process.env.HOST to the endpoint host for my instance and set process.env.AWS_ACCESS_KEY_ID and process.env.AWS_SECRET_ACCESS_KEY to my secret key and access key and I run that script, I get:

The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.

The Canonical String for this request should have been
'POST
/%2A/%2A/_search/

content-length:26
content-type:application/json
host:search-dave-6jy2cskdfaye4ji6gfa6x375ve.us-west-2.es.amazonaws.com
x-amz-date:20151216T035010Z

content-length;content-type;host;x-amz-date
baa6846b65b050d71831bb2e4cd6e6f1593902f6d82b16a6c1f9979d14cfcd12'

The String-to-Sign should have been
'AWS4-HMAC-SHA256
20151216T035010Z
20151216/us-west-2/es/aws4_request
7b030af77da5871723b5be5026410d2a968891df84263d7a6532acd5cc83d9d4'

Note that the asterisks in the request path were replaced by %2A in their Canonical String. So in order to properly sign requests to paths with asterisks, AWS4 needs to encode those asterisks as %2A. This PR does that. With this change, my script logs data from my instance. Thanks for your attention!

[mhart]

Apologies for letting this sit for a few days. It looks good! I'm surprised that Amazon didn't include this in its test suite: http://docs.aws.amazon.com/general/latest/gr/signature-v4-test-suite.html

I'm going to check if there's any other characters that might've slipped through, or whether it's just the asterisk.

[orasimus]

There is at least : which should be replaced with %3A.

Noticed this today - I'm using : as a part of the path in my API endpoints.

[mhart]

Apologies for still not getting this in – I'm wanting to do it right and have discovered a whooooooole rabbit hole of issues with the way that AWS services deal with their encoding, from incorrect URI encoding on certain characters, to odd query string handling, to completely incorrect documentation and test suites (which the SDKs seem to completely ignore btw).

I'm hoping to resolve this in the next few days – basically I think I'll aim to match what the services are expecting instead of following the documentation and test suite, but it means dealing with some hairy edge cases.

[mhart]

Thanks so much for raising this – I ended up finding out so much more about the various AWS quirks than I wanted to – especially when it comes to S3, ugh...

Anyway, it should all be fixed by 2901b5a which has been published as v1.2.0

Thanks again!