S3 api is missing both getSignedUrl and createPresignedPost

[justinmchase]

Here's the link to the javascript api for createPresignedPost.

Its possible that these functions in javascript skd have been manually added these since they're not actually api endpoints. Is there some utility class in here where I could effectively sign urls still?

For context, in case you're not aware, these two singing apis will create a url with an encrypted token in it which you can then hand off to someone else, including a browser, and it can then be used to fetch or upload a file directly from the browser. This is how you'd manage access to private buckets and also its a pretty slick way to handle file uploads without having to go through your api server at all.

[justinmchase]

It looks like the AWSSignerV4 might cover it?

const signer = new AWSSignerV4();
const body = new TextEncoder().encode("Hello World!")
const request = new Request("https://test-bucket.s3.amazonaws.com/test", {
  method: "PUT",
  headers: { "content-length": body.length.toString() },
  body,
});
return await signer.sign("s3", request);

[justinmchase]

It looks like its not quite going to work though its close...

Down in the sign function it has:

const payloadHash = sha256(body ?? new Uint8Array()).hex();
if (service === 's3') {
  headers.set("x-amz-content-sha256", payloadHash);
}

This assumes that a body is provided. In these cases you need to be able to not include the body as part of the signature, because you can't know what the body is in this case. You're giving them a blank check basically to upload whatever they want. I will have the contentType and the contentLength but not the actual content.

Here is the generated code in the node aws-sdk:

function createPresignedPost(params, callback) {
    if (typeof params === 'function' && callback === undefined) {
      callback = params;
      params = null;
    }

    params = AWS.util.copy(params || {});
    var boundParams = this.config.params || {};
    var bucket = params.Bucket || boundParams.Bucket,
      self = this,
      config = this.config,
      endpoint = AWS.util.copy(this.endpoint);
    if (!config.s3BucketEndpoint) {
      endpoint.pathname = '/' + bucket;
    }

    function finalizePost() {
      return {
        url: AWS.util.urlFormat(endpoint),
        fields: self.preparePostFields(
          config.credentials,
          config.region,
          bucket,
          params.Fields,
          params.Conditions,
          params.Expires
        )
      };
    }

    if (callback) {
      config.getCredentials(function (err) {
        if (err) {
          callback(err);
        } else {
          try {
            callback(null, finalizePost());
          } catch (err) {
            callback(err);
          }
        }
      });
    } else {
      return finalizePost();
    }
  }

function preparePostFields(
  credentials,
  region,
  bucket,
  fields,
  conditions,
  expiresInSeconds
) {
  var now = this.getSkewCorrectedDate();
  if (!credentials || !region || !bucket) {
    throw new Error('Unable to create a POST object policy without a bucket,'
      + ' region, and credentials');
  }
  fields = AWS.util.copy(fields || {});
  conditions = (conditions || []).slice(0);
  expiresInSeconds = expiresInSeconds || 3600;

  var signingDate = AWS.util.date.iso8601(now).replace(/[:\-]|\.\d{3}/g, '');
  var shortDate = signingDate.substr(0, 8);
  var scope = v4Credentials.createScope(shortDate, region, 's3');
  var credential = credentials.accessKeyId + '/' + scope;

  fields['bucket'] = bucket;
  fields['X-Amz-Algorithm'] = 'AWS4-HMAC-SHA256';
  fields['X-Amz-Credential'] = credential;
  fields['X-Amz-Date'] = signingDate;
  if (credentials.sessionToken) {
    fields['X-Amz-Security-Token'] = credentials.sessionToken;
  }
  for (var field in fields) {
    if (fields.hasOwnProperty(field)) {
      var condition = {};
      condition[field] = fields[field];
      conditions.push(condition);
    }
  }

  fields.Policy = this.preparePostPolicy(
    new Date(now.valueOf() + expiresInSeconds * 1000),
    conditions
  );
  fields['X-Amz-Signature'] = AWS.util.crypto.hmac(
    v4Credentials.getSigningKey(credentials, shortDate, region, 's3', true),
    fields.Policy,
    'hex'
  );

  return fields;
}

function preparePostPolicy(expiration, conditions) {
  return AWS.util.base64.encode(JSON.stringify({
    expiration: AWS.util.date.iso8601(expiration),
    conditions: conditions
  }));
}

[danopia]

Thanks for the report. As you noted, presigned URLs aren't actually an API call and thus weren't in the scope of this API client codegen effort. I can see the usefulness though and it would make sense to expose the necessary aspects + include an example of making a presigned URL for S3.

[justinmchase]

Do you have any idea of a work around? I'm blocked so hard on this and I cannot figure it out. Presigned URL's are a core feature and I can't seem to unwind their horrible code into a simple function. I'll have to abandon Deno just so I can use the amazon sdk.

All 3 of the Deno projects for the amazon SDK have this same bug where they're generating code off of the json definitions and lack the presigned url apis, its a real bummer.

[danopia]

You can use the real full-fat SDK to presign URLs today, as long as you're comfortable with the flags the main port needs (--unstable --allow-read --allow-env)

import { getSignedUrl } from "https://deno.land/x/[email protected]/s3-request-presigner/mod.ts";
import { S3Client } from "https://deno.land/x/[email protected]/client-s3/S3Client.ts";
import { GetObjectCommand } from "https://deno.land/x/[email protected]/client-s3/commands/GetObjectCommand.ts";
// set the credentials
const client = new S3Client({
  region: "ap-south-1",
  credentials: {
    accessKeyId: 'AKIAANDSOON',
    secretAccessKey: 'thisismysecret',
  },
});
// build the command to presign
const command = new GetObjectCommand({
  Bucket: 'my-bucket',
  Key: 'my/key/is/here',
});
const url = await getSignedUrl(client, command, { expiresIn: 3600 });

[justinmchase]

Awesome, I was just trying this out too so its good to see.

Though now that I have the URL I cannot seem to figure out how to use it via curl.

I had this working via the v2 api last time I went to do this but it seems like its different now and its not clear why it doesn't work :(

[justinmchase]

echo "testing 123" > hello.txt
URL=$(deno run --unstable -A main.ts)
curl "$URL" -T hello.txt

I'm using minio and so I had to add an endpoint and forcePathStyle

import { getSignedUrl } from "https://deno.land/x/[email protected]/s3-request-presigner/mod.ts";
import { S3Client } from "https://deno.land/x/[email protected]/client-s3/S3Client.ts";
import { PutObjectCommand } from "https://deno.land/x/[email protected]/client-s3/commands/PutObjectCommand.ts";
// set the credentials
const client = new S3Client({
  region: "us-east-1",
  endpoint: "http://localhost:9000",
  forcePathStyle: true,
  credentials: {
    accessKeyId: 'AjAOk2gNRU',
    secretAccessKey: 'Wk1HVyV8WP2Nh3O9QfLvTW9dOwR0ysqthZrP2Smf',
  },
});
// build the command to presign
const command = new PutObjectCommand({
  Bucket: 'uploads',
  Key: 'test123',
});
const url = await getSignedUrl(client, command, { expiresIn: 3600 });
console.log(url)

All of the other apis work so the creds are right its just somehow this getSignedUrl is doing something its not expecting that or I have to add some headers that I don't know about, you didn't have to do that in the v2 api...

[danopia]

Good to hear you got somewhere with the official SDK. I would still consider this in-scope to add in this repository somewhere, but I'll let this stay closed unless someone wants to revive the feature request.

[dansalias]

In case it's useful as a reference - I've published a Deno module specifically for creating S3 presigned urls: https://deno.land/x/[email protected].

Here you can see how S3 presigned URLs relate to signatures: https://github.com/dansalias/aws_s3_presign/blob/trunk/mod.ts#L102-L111.

[danopia]

Thanks, that looks like a pretty clean and tidy module for anyone who wants to specifically presign S3 URLs!

Given that this codebase is pretty married to a signingFetcher interface which conflates both tasks, I don't think I'll be able to offer any code nearly as concise in /x/aws_api. (Having signingFetcher fetch without signing was pretty workable but signing without a fetch doesn't fit into the types without some refactor. If I eventually work presigning into /x/aws_api it would handle some extra goodies like path-style routing & other AWS partitions, so there'd still be some benefit I suppose.)

Until that happens I'd recommend your /x/aws_s3_presign to anyone else with this usecase. Brief example of using both libraries together:

import {
  DefaultCredentialsProvider,
  getDefaultRegion,
} from "https://deno.land/x/[email protected]/client/credentials.ts";
import {
  getSignedUrl,
} from "https://deno.land/x/[email protected]/mod.ts";

async function presignGetObject(bucket: string, key: string) {
  const credentials = await DefaultCredentialsProvider.getCredentials();
  return getSignedUrl({
    accessKeyId: credentials.awsAccessKeyId,
    secretAccessKey: credentials.awsSecretKey,
    sessionToken: credentials.sessionToken,
    region: credentials.region ?? getDefaultRegion(),

    bucketName: bucket,
    objectPath: `/${key}`,
  });
}

console.log(await presignGetObject('my-bucket', 'my-key'));

This way the credential loading is consistent with the rest of the application.

[dansalias]

Perfect, thanks for the example. I'm sure it'll prove useful for others. And great work on the Deno ports so far!

[yogesnsamy]

@dansalias Currently there's an error in using your module. Could you please have a look at this PR: dansalias/aws_s3_presign#4

[danopia]

🚀 There's now a basic presigner in v0.8.1. It's similar to /x/aws_s3_presign except it uses /x/aws_api's credential fetching and request signing. This presigner is thus async (returns a Promise).

Two different ways of using:

1. AWSSignerV4 offers presigning given a full URL and this is pretty straightforward but you have to construct the signer with credentials yourself.

import { DefaultCredentialsProvider } from "https://deno.land/x/[email protected]/client/credentials.ts";
import { AWSSignerV4 } from "https://deno.land/x/[email protected]/client/signing.ts";

const credentials = await DefaultCredentialsProvider.getCredentials();
const signer = new AWSSignerV4('us-east-2', credentials);

const url = await signer.presign('s3', {
  method: 'GET',
  url: 'https://my-bucket.s3.amazonaws.com/my-key',
});

3. New module /extras/s3-presign.ts adds S3-specific presigning logic and constructs credentials and endpoints automatically.

import { getPresignedUrl } from "https://deno.land/x/[email protected]/extras/s3-presign.ts";

const url = await getPresignedUrl({
  region: 'us-east-2',
  bucket: 'my-bucket',
  path: '/my-key',
});

[dansalias]

@danopia super useful, thanks for the update!

[randallb]

One other note: I don't think it's possible to sign headers, etc. I'm looking to add metadata to the request, and using the integrated signer, it doesn't look like it supports this. Would this be something folks would be open to me contributing?