(aws-s3-deployment): empty files are uploaded to bucket

[I2olanD]

What is the problem?

When i upload my locally build dist, files are uploaded with 0 bytes (empty files).

Reproduction Steps

Use s3-deployment to upload local assets.

  new s3deploy.BucketDeployment(this, 'DeployWithInvalidation', {
      sources: [s3deploy.Source.asset('../packages/some-dir/dist')],
      destinationBucket: bucket,
      distribution,
    });

What did you expect to happen?

Files are uploaded correctly

What actually happened?

Files are empty

CDK CLI Version

1.132.0

Framework Version

No response

Node.js Version

16.14.0

OS

MacOS 12.2.1

Language

Typescript

Language Version

Typescript 3.9.7

Other information

_No response

_

[NGL321]

Hey @I2olanD,

Thank you for reporting this, but I have been unable to reproduce the behavior. Could you please provide more context so we can properly evaluate this problem?
This may have to do with the files you are attempting to upload as assets, so it would be best if you could confirm this behavior occurs for you even outside of the original context. Please share the results and we can look further into this.

[I2olanD]

Hi @NGL321,

thanks for the reply! I migrated to v2 and the problem is gone now. But what worked for me before migrating was deleting cdk.out folder and remove all the assets from S3.

Best

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[0xdevalias]

I'm also seeing this on CDK 1.x; I believe I first saw it on the very old version this project was on (1.32.2), and am still seeing it on the latest 1.x (1.201.0) as part of my process of upgrading to 2.x.

I haven't looked deeper into trying to debug the why of this yet. Will update this issue with details if I find anything useful.

Potentially related:

• lambda: corrupt zip archive asset produced when using node v15.6 #12536
• (aws-lambda-nodejs): Uploaded file must be a non-empty zip #18459

[0xdevalias]

So I basically followed the advice in this comment as the basis for further exploration:

As others have stated, clearing the staging bucket of affected assets is required to fix this problem. After becoming aware of this issue, running cdk deploy ... -vvv indicated that CDK was detecting a pre-existing asset within the staging bucket. With this asset already uploaded, CDK skipped uploading the current copy (leaving a corrupted zip file in place) and proceeded to attempt to use the corrupted zip file in the following Lambda function deployment.

My issue was resolved after updating to the previously mentioned Node and CDK versions, then removing the assets which were indicated in the verbose output produced by the -vvv flag.

Originally posted by @bgshacklett in #12536 (comment)

I basically deployed my stack with --exclusively --force -vvv, looked through the generated CloudFormation template json in cdk.out to identify the asset's that corresponded with my BucketDeployment, then looked for those asset hashes in the output from the deploy to see what was happening.

From there I noticed that it basically wasn't trying to generate/upload the new asset zip's because it could already see that an existing zip existed in the staging bucket. Because that zip was corrupted, it had the effect of never replacing the corrupted zip file.

The solution was to delete the corrupted zip from the staging bucket and local caches, then run the deploy again with --force to ensure it would upload the (now missing) zip file.

Obviously this doesn't identify how the corrupted zip was created/uploaded in the first place, but at least it gives enough detail that it would be possible to trace the logic and ensure that old cached files in the staging bucket are at least invalidated properly so that the newly built assets are uploaded in their place.

Full Writeup

(🔒 PrivateRef)

Within cdk.out/REDACTED.template.json, we find the entries that correspond to our BucketDeployment:

We find this in DeployCustomResource218AF6A4 ("Type": "Custom::CDKBucketDeployment"), which tells us the asset parameter refs for the SourceBucketNames, SourceObjectKeys, etc; as well as the DestinationBucketName ref (SiteBucket397A1860):

• DeployCustomResource218AF6A4
  • SourceBucketNames
    • AssetParameters615e72591af2c9b172c96cabcf84169d6275f06463d0bb3d11cdf9121718510fS3Bucket0F4B11F8
    • AssetParameters1ebc9d3ac2033816c4abb63e4afd69d350b4aba8704cc9236b82ea520b74f4b0S3Bucket5EA66AEF
  • SourceObjectKeys
    • AssetParameters615e72591af2c9b172c96cabcf84169d6275f06463d0bb3d11cdf9121718510fS3VersionKey8BFEEF5B
    • AssetParameters1ebc9d3ac2033816c4abb63e4afd69d350b4aba8704cc9236b82ea520b74f4b0S3VersionKeyD171B821
  • DestinationBucketName
    • SiteBucket397A1860
      • "BucketName": "app.dev.REDACTED"

We can then look at the stack's Parameters to look up the parameter refs:

• Parameters
  • AssetParameters615e72591af2c9b172c96cabcf84169d6275f06463d0bb3d11cdf9121718510fS3Bucket0F4B11F8
    • S3 bucket for asset \"615e72591af2c9b172c96cabcf84169d6275f06463d0bb3d11cdf9121718510f\"
  • AssetParameters615e72591af2c9b172c96cabcf84169d6275f06463d0bb3d11cdf9121718510fS3VersionKey8BFEEF5B
    • S3 key for asset version \"615e72591af2c9b172c96cabcf84169d6275f06463d0bb3d11cdf9121718510f\"
  • AssetParameters1ebc9d3ac2033816c4abb63e4afd69d350b4aba8704cc9236b82ea520b74f4b0S3Bucket5EA66AEF
    • S3 bucket for asset \"1ebc9d3ac2033816c4abb63e4afd69d350b4aba8704cc9236b82ea520b74f4b0\"
  • AssetParameters1ebc9d3ac2033816c4abb63e4afd69d350b4aba8704cc9236b82ea520b74f4b0S3VersionKeyD171B821
    • S3 key for asset version \"1ebc9d3ac2033816c4abb63e4afd69d350b4aba8704cc9236b82ea520b74f4b0\"

From those Parameters we can see that there are 2 underlying assets we're working with, and based on the order of the assets listed in the code in the REDACTED stack CDK infrastructure code, we can infer which asset refers to which path:

• 615e72591af2c9b172c96cabcf84169d6275f06463d0bb3d11cdf9121718510f: Source.asset('../REDACTED1/build')
• 1ebc9d3ac2033816c4abb63e4afd69d350b4aba8704cc9236b82ea520b74f4b0: Source.asset('../REDACTED2/dist')

Now that we know the asset hashes we're looking for, and what they correlate to, we can go back and look at the output from yarn cdk-dev deploy REDACTED --exclusively --force -vvv and look for relevant details for those assets.

Searching for asset 615e72591af2c9b172c96cabcf84169d6275f06463d0bb3d11cdf9121718510f we find the following snippets:

Preparing asset 615e72591af2c9b172c96cabcf84169d6275f06463d0bb3d11cdf9121718510f: {"path":"asset.615e72591af2c9b172c96cabcf84169d6275f06463d0bb3d11cdf9121718510f","id":"615e72591af2c9b172c96cabcf84169d6275f06463d0bb3d11cdf9121718510f","packaging":"zip","sourceHash":"615e72591af2c9b172c96cabcf84169d6275f06463d0bb3d11cdf9121718510f","s3BucketParameter":"AssetParameters615e72591af2c9b172c96cabcf84169d6275f06463d0bb3d11cdf9121718510fS3Bucket0F4B11F8","s3KeyParameter":"AssetParameters615e72591af2c9b172c96cabcf84169d6275f06463d0bb3d11cdf9121718510fS3VersionKey8BFEEF5B","artifactHashParameter":"AssetParameters615e72591af2c9b172c96cabcf84169d6275f06463d0bb3d11cdf9121718510fArtifactHash9E10881F"}

Storing asset asset.615e72591af2c9b172c96cabcf84169d6275f06463d0bb3d11cdf9121718510f at s3://cdktoolkit-stagingbucket-REDACTED/assets/615e72591af2c9b172c96cabcf84169d6275f06463d0bb3d11cdf9121718510f.zip

[0%] start: Publishing 615e72591af2c9b172c96cabcf84169d6275f06463d0bb3d11cdf9121718510f:current

[0%] check: Check s3://cdktoolkit-stagingbucket-REDACTED/assets/615e72591af2c9b172c96cabcf84169d6275f06463d0bb3d11cdf9121718510f.zip

[AWS s3 200 0.843s 0 retries] listObjectsV2({
  Bucket: 'cdktoolkit-stagingbucket-REDACTED',
  Prefix: 'assets/615e72591af2c9b172c96cabcf84169d6275f06463d0bb3d11cdf9121718510f.zip',
  MaxKeys: 1
})

[20%] found: Found s3://cdktoolkit-stagingbucket-REDACTED/assets/615e72591af2c9b172c96cabcf84169d6275f06463d0bb3d11cdf9121718510f.zip

[40%] success: Published 615e72591af2c9b172c96cabcf84169d6275f06463d0bb3d11cdf9121718510f:current

From this, we can see that it seems to have found an existing asset in the cdktoolkit-stagingbucket-REDACTED, and so decided that it didn't need to re-upload anything. This means that so long as that same hashed file is present in the staging bucket, and our code generates that same hash, the old file will always remain, which if it's corrupted, means it will always stay corrupted.

Based on the above verbose output, we can now go and download the 2 existing asset files from cdktoolkit-stagingbucket-REDACTED to confirm our theories:

• s3://cdktoolkit-stagingbucket-REDACTED/assets/615e72591af2c9b172c96cabcf84169d6275f06463d0bb3d11cdf9121718510f.zip
• s3://cdktoolkit-stagingbucket-REDACTED/assets/1ebc9d3ac2033816c4abb63e4afd69d350b4aba8704cc9236b82ea520b74f4b0.zip

We can do this as follows:

⇒ yarn awsvault-dev aws s3 cp s3://cdktoolkit-stagingbucket-REDACTED/assets/615e72591af2c9b172c96cabcf84169d6275f06463d0bb3d11cdf9121718510f.zip ~/Desktop

⇒ yarn awsvault-dev aws s3 cp s3://cdktoolkit-stagingbucket-REDACTED/assets/1ebc9d3ac2033816c4abb63e4afd69d350b4aba8704cc9236b82ea520b74f4b0.zip ~/Desktop

Opening the files we downloaded to the desktop, we can see that:

• 615e72591af2c9b172c96cabcf84169d6275f06463d0bb3d11cdf9121718510f.zip contains our empty / 0 byte REDACTED1 assets
• 1ebc9d3ac2033816c4abb63e4afd69d350b4aba8704cc9236b82ea520b74f4b0.zip is an empty zip file (since we never built the code for REDACTED2)

Since we have now confirmed that the uploaded asset for 615e72591af2c9b172c96cabcf84169d6275f06463d0bb3d11cdf9121718510f.zip is corrupted, and won't be re-uploaded as the AWS CDK framework see's it already exists, we will need to remove the corrupted file to fix this issue.

We can do that in a similar to how we downloaded the file above, but this time, removing it from the cdktoolkit-stagingbucket-REDACTED bucket:

⇒ yarn awsvault-dev aws s3 rm s3://cdktoolkit-stagingbucket-REDACTED/assets/615e72591af2c9b172c96cabcf84169d6275f06463d0bb3d11cdf9121718510f.zip

We also probably want to remove any staged/cached assets in cdk.out and cdk.out/.cache/:

⇒ rm -rf cdk.out/asset.615e72591af2c9b172c96cabcf84169d6275f06463d0bb3d11cdf9121718510f

⇒ rm -rf cdk.out/.cache/615e72591af2c9b172c96cabcf84169d6275f06463d0bb3d11cdf9121718510f.zip
# Note: This file didn't actually exist in the local .cache

Once we have cleaned up the corrupted asset(s), we can try re-deploying.

Unfortunately the CDK diff doesn't show us anything about our built assets changing, so we'll just jump straight to a yarn cdk-dev deploy REDACTED --exclusively -vvv and see what happens:

The relevant snippets from the output are:

Preparing asset 615e72591af2c9b172c96cabcf84169d6275f06463d0bb3d11cdf9121718510f: {"path":"asset.615e72591af2c9b172c96cabcf84169d6275f06463d0bb3d11cdf9121718510f","id":"615e72591af2c9b172c96cabcf84169d6275f06463d0bb3d11cdf9121718510f","packaging":"zip","sourceHash":"615e72591af2c9b172c96cabcf84169d6275f06463d0bb3d11cdf9121718510f","s3BucketParameter":"AssetParameters615e72591af2c9b172c96cabcf84169d6275f06463d0bb3d11cdf9121718510fS3Bucket0F4B11F8","s3KeyParameter":"AssetParameters615e72591af2c9b172c96cabcf84169d6275f06463d0bb3d11cdf9121718510fS3VersionKey8BFEEF5B","artifactHashParameter":"AssetParameters615e72591af2c9b172c96cabcf84169d6275f06463d0bb3d11cdf9121718510fArtifactHash9E10881F"}

Storing asset asset.615e72591af2c9b172c96cabcf84169d6275f06463d0bb3d11cdf9121718510f at s3://cdktoolkit-stagingbucket-REDACTED/assets/615e72591af2c9b172c96cabcf84169d6275f06463d0bb3d11cdf9121718510f.zip

But despite that, looking at the S3 bucket, neither of those files were actually uploaded.. so I guess we'll need to try again with --force (yarn cdk-dev deploy REDACTED --exclusively --force -vvv):

The relevant snippets from the output this time are:

Preparing asset 615e72591af2c9b172c96cabcf84169d6275f06463d0bb3d11cdf9121718510f: {"path":"asset.615e72591af2c9b172c96cabcf84169d6275f06463d0bb3d11cdf9121718510f","id":"615e72591af2c9b172c96cabcf84169d6275f06463d0bb3d11cdf9121718510f","packaging":"zip","sourceHash":"615e72591af2c9b172c96cabcf84169d6275f06463d0bb3d11cdf9121718510f","s3BucketParameter":"AssetParameters615e72591af2c9b172c96cabcf84169d6275f06463d0bb3d11cdf9121718510fS3Bucket0F4B11F8","s3KeyParameter":"AssetParameters615e72591af2c9b172c96cabcf84169d6275f06463d0bb3d11cdf9121718510fS3VersionKey8BFEEF5B","artifactHashParameter":"AssetParameters615e72591af2c9b172c96cabcf84169d6275f06463d0bb3d11cdf9121718510fArtifactHash9E10881F"}

Storing asset asset.615e72591af2c9b172c96cabcf84169d6275f06463d0bb3d11cdf9121718510f at s3://cdktoolkit-stagingbucket-REDACTED/assets/615e72591af2c9b172c96cabcf84169d6275f06463d0bb3d11cdf9121718510f.zip

[0%] start: Publishing 615e72591af2c9b172c96cabcf84169d6275f06463d0bb3d11cdf9121718510f:current

[0%] check: Check s3://cdktoolkit-stagingbucket-REDACTED/assets/615e72591af2c9b172c96cabcf84169d6275f06463d0bb3d11cdf9121718510f.zip

[AWS s3 200 0.841s 0 retries] listObjectsV2({
  Bucket: 'cdktoolkit-stagingbucket-REDACTED',
  Prefix: 'assets/615e72591af2c9b172c96cabcf84169d6275f06463d0bb3d11cdf9121718510f.zip',
  MaxKeys: 1
})

[60%] build: Zip /Users/devalias/dev/REDACTED/cdk.out/asset.615e72591af2c9b172c96cabcf84169d6275f06463d0bb3d11cdf9121718510f -> cdk.out/.cache/615e72591af2c9b172c96cabcf84169d6275f06463d0bb3d11cdf9121718510f.zip

[60%] upload: Upload s3://cdktoolkit-stagingbucket-REDACTED/assets/615e72591af2c9b172c96cabcf84169d6275f06463d0bb3d11cdf9121718510f.zip

[AWS s3 200 5.956s 0 retries] putObject({
  Body: <Buffer REDACTED ... 603949 more bytes>,
  Bucket: 'cdktoolkit-stagingbucket-REDACTED',
  Key: 'assets/615e72591af2c9b172c96cabcf84169d6275f06463d0bb3d11cdf9121718510f.zip',
  ContentType: 'application/zip',
  ServerSideEncryption: 'aws:kms',
  SSEKMSKeyId: undefined
})

[100%] success: Published 615e72591af2c9b172c96cabcf84169d6275f06463d0bb3d11cdf9121718510f:current

As we can see in the above output, this time AWS CDK actually built the relevant zip files, and then uploaded them to the S3 bucket.

And then re-downloading the zip file to check if it's contents are empty (0 bytes) still:

⇒ yarn awsvault-dev aws s3 cp s3://cdktoolkit-stagingbucket-REDACTED/assets/615e72591af2c9b172c96cabcf84169d6275f06463d0bb3d11cdf9121718510f.zip ~/Desktop

Which we can see, the files are correctly non-zero bytes now: