Cant upload to s3

[allanhortle]

Uploading files to s3 causes this error:

Error: write EPIPE
  at errnoException (net.js:904:11)
  at Object.afterWrite (net.js:720:19)

It's a pretty uninformative error so if you need any other details let me know :)

[grabbou]

We are refactoring that module and others in #526. I will fix that issue later today for you (if I will able to reproduce it) and the fix will be present in new version of Amazon File module as well.

[allanhortle]

Hooray! Just got started with Keystone. It's pretty much exactly what i've been looking for. Props for an awesome product.

[grabbou]

Going to review that today.

[allanhortle]

With 0.2.26 it will think it's uploaded the file and list a link in the admin page. But in reality it hasn't actually done anything. For what it's worth I am uploading to ap-southeast-2

[23limes]

Just upgraded from 0.2.22 to 0.2.26 neither of these versions actually uploads to S3, just as @allanhortle said, it thinks it uploaded a file and lists the link, but nothing actually shows up in my S3 bucket. I am in the us-west-1 region. Any insight would be greatly appreciated.

[aknoerig]

Any progress on this? I found a note in the release history about an S3_REGION env variable. This makes the generated URL look correct, but it still doesn't seem to upload anything. Maybe it has to do with my setup on the Amazon side, which is a bit confusing, with permissions, CORS, IAM users, and all. Maybe someone who has got it working can provide their configuration? Thanks!

[23limes]

It turned out to be a permissions problem on our s3 bucket - hard to debug, but re-creating the bucket with correct permissions did the trick for us.

So, in S3 console, grant Everyone "List", and Authenticated Users all 4 permissions.
Then this is our CORS setting for testing:

[CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"]
[CORSRule]
[AllowedOrigin]put origin or astrisk here[/AllowedOrigin]
[AllowedMethod]GET[/AllowedMethod]
[AllowedMethod]POST[/AllowedMethod]
[AllowedMethod]PUT[/AllowedMethod]
[AllowedHeader]*[/AllowedHeader]
[/CORSRule]
[/CORSConfiguration]

Note: Change square brackets to angle brackets (won't let me post with angle brackets)
Hope that helps!

[aknoerig]

Thanks, that's the setup I have. I re-created it, but no luck yet. It might have to do with the user credentials, but I'm not sure how to debug this as keystone doesn't seem to throw any errors.

[reubano]

Not working for eu-central-1 region either.

[morenoh149]

some s3 related bug was fixed recently keystonejs/keystone@b7c4cf1 have you all had a chance to test that?

[reubano]

No, but is that specifically related to eu regions? The standard US region works fine.

[morenoh149]

@reubano ah it isn't specific to eu region.

[JedWatson]

Hey guys, can someone confirm whether they are still able to reproduce this? As @morenoh149 mentions we fixed an S3 bug recently, and we can't replicate it now.

[aknoerig]

Hmm, I currently can't get it to work in either region -- so probably a misconfiguration on my side. I just don't know how to debug this further, since I'm not getting any error message.
The file is not uploaded, yet the keystone UI shows a link to the file on S3. When I follow it, I'm of course getting:
<Code>NoSuchKey</Code>
<Message>The specified key does not exist.</Message>

Any pointers on how to debug this further? It seems like keystone is using a plugin called knox to communicate with S3. Are its error messages passed back somewhere to the UI or console?

[brianjd]

@aknoerig You can always edit s3file.js and create your own inspection points in the doUpload(). My suggestion is to inspect field.s3config and insure that it looks proper. You can do some additional information gathering in the callback itself to help isolate what is happening in your situation. Hope this helps.

[morenoh149]

@aknoerig that sounds a lot like what this fix solved for me. keystonejs/keystone@b7c4cf1 to debug try https://github.com/node-inspector/node-inspector and dig in around https://github.com/keystonejs/keystone/blob/master/lib/fieldTypes/s3file.js#L258

[aknoerig]

Thanks guys! I can now see that I'm getting a 403 on knox.createClient(), so it's probably a credentials problem. Why does Amazon have to make this so difficult..

[aknoerig]

Btw, the bug reported here nevertheless still seems to hold, as knox' readme says:

Convenience APIs such as putFile and putStream currently do not work as expected with buckets in regions other than US Standard without explicitly specify the region option.

And putFile is what keystone is using.

[webteckie]

@aknoerig does the debugging you did on this leads to improving a keystone code path so that it logs that error to avoid someone else go through the same pain?

[weekwood]

ok @aknoerig , I found issue for 403

knox.createClient(field.s3config).putFile(file.path, path + name, {
            'Content-Type': file.mimetype,
            'x-amz-acl': 'public-read'
        }, function(err, res) {

seems mimetype is undefined. if you make it right or just remove, everything works fine.

I will send pull request later today to fix this.

[brianjd]

@weekwood , @morenoh149 already submitted one.

[weekwood]

@brianjd #780 ?

[morenoh149]

@weekwood yes. This issue was not closed earlier because @reubano reported it may be related to aws region settings. I'm going to close this issue. @reubano if you run into that bug again after keystone 0.3.0 is released feel free to open a new issue with reproducable code.

Though I agree with @webteckie that keystone should provide a better error message in this case.

[awinecki]

Hey there, not sure is this is the right issue, however I feel there's still some problem with EU buckets.

I'm running 0.3.2 now, and here's my fields/types/s3file/S3FileType.js contents (relevant lines):

        console.log(file.mimetype);
267     knox.createClient(field.s3config).putFile(file.path, path + filename, {
            'Content-Type': file.mimetype,
            'x-amz-acl': 'public-read'
        }, function(err, res) {

The filetype I'm trying to upload is video/mp4. The console.log returns video/mp4 correctly, however the upload still crashes with 403 response from Amazon.

[morenoh149]

@awinecki please read https://github.com/LearnBoost/knox#endpoint and region. We are using putFile which only seems to work for US standard. Could you add support for specifying the region by the developer?