CustomResource Lambda with wrong permissions depending on the umask

[ilkomiliev]

description of the bug:

we have very permissive umask set on our linux EC2 instances - 0027
with this umask in place the default permissions are too restrictive for "normal" Lambda deployment that's why for our lambda functions we apply special mechanism to set them properly to 0755. Unfortunately the generated lambda function for the AwsCustomResource doesn't do any extra permission settings and takes them as they are set on the file system. Thus trying to deploy stack containing Custom Resource leads to the following error:

2020-05-27T12:02:51.288Z	undefined	ERROR	Uncaught Exception 	
{
    "errorType": "Error",
    "errorMessage": "EACCES: permission denied, open '/var/task/index.js'",
    "code": "EACCES",
    "errno": -13,
    "syscall": "open",
    "path": "/var/task/index.js",
    "stack": [
        "Error: EACCES: permission denied, open '/var/task/index.js'",
        "    at Object.openSync (fs.js:458:3)",
        "    at Object.readFileSync (fs.js:360:35)",
        "    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1152:22)",
        "    at Module.load (internal/modules/cjs/loader.js:977:32)",
        "    at Function.Module._load (internal/modules/cjs/loader.js:877:14)",
        "    at Module.require (internal/modules/cjs/loader.js:1019:19)",
        "    at require (internal/modules/cjs/helpers.js:77:18)",
        "    at _tryRequire (/var/runtime/UserFunction.js:75:12)",
        "    at _loadUserApp (/var/runtime/UserFunction.js:95:12)",
        "    at Object.module.exports.load (/var/runtime/UserFunction.js:140:17)"
    ]
}

Reproduction Steps

on a Linux based OS set the umask to 0027
try to deploy a stack containing custom resource (directly or indirectly). One of the standard CDK methods we found out to cause the problem is registering Cognito Custom Domain, i.e.:

target = r53.RecordTarget.from_alias(r53_targets.UserPoolDomainTarget(user_pool_domain))
public_zone = fr53.import_hosted_zone(scope, 'hzpublic', HostedZoneTypesEnum.PUBLIC)
r53.ARecord(
            scope, 
            id='r53rec', 
            target=target, 
            zone=public_zone, 
            comment='Cognito CloudFront Alias', 
            record_name=self.cognito_config.domain_name
)

But of course anything like this also will fail:

cr.AwsCustomResource(
            scope,
            r_id,
policy=cr.AwsCustomResourcePolicy.from_sdk_calls(resources=cr.AwsCustomResourcePolicy.ANY_RESOURCE),
            on_update=call
        )

Ensure that the lambda function is really created at this time and not cached from previous executions (umask is applied at the time the file or dir is created!) - it can be found on the assets bucket in S3, check the modification dates.

Download the zip file as found in S3 and check the permissions on the files:

35d0a3ea655835ce2bf399c19e80a38397cebc9cff491b04a9312c92d338669.zip
-rw-r----- 1 xxx xxx   336 Jan  1  1980 index.d.ts
-rw-r----- 1 xxx xxx 20679 Jan  1  1980 index.js

Error Log

1. the permissions of the files in the zip are wrong

2. in CloudWatch you can find the above error repeated several times

Environment

• CLI Version : 1.41.0 (build 9e071d2)
• Framework Version:
• OS : Amazon Linux custom setup
• Language : python for CDK development / JS for the lambda

Other

n.a.

This is 🐛 Bug Report

[ilkomiliev]

Hi,
any info about this - what I'd like to know is, if this is really CDK issue or CF - does the CDK handle the generation of the lambda function or it just delegates this to CF? If the latter we are covered by Enterprise support and I meanwhile pushed this also there - the internal case ID is 7033616551, if someone can take a look there also. The workaround currently in place is very ugly:

• fail once
• download the zip file and change the permissions
• upload it back to s3
• hope that it doesn't change too often
• :-(

Thanks!

[Aerixs]

I have the same issue when a custom resource Lambda (from FargateCluster) gets deployed from Linux with a umask of 0022.

As soon as the zip arrives in s3 it has the wrong permissions (0640). On the filesystem the permissions of the files are 0644.

File permissions when deploying from Windows are set to 0666.

[ilkomiliev]

This is strange, because 0022 was the umask, which we previously had and the permissions were ok with it, so perhaps this depends on something else.

[amzcarlli]

Another occurrence of permission set differently once is uploaded to S3.
internal case id #7079559831

Environment
CLI Version : 1.42.1

permissions on index.js
-rwxr-xr-x+ 1 Domain Users 1786 Jun 8 22:00 index.js

after cdk synth and zipped code uploaded to S3
-rwx------+ 1 Domain Users 1760 Jun 8 20:10 index.js

[educlos]

Similar issue here, the files are 0755 on the machine and end up 0640 once on the zip file on s3.
What we noted on our side however is that the issue is occurring when running the cdk from an EC2 ubuntu instance, but not when following the exact same steps and using the same credentials on a physical ubuntu laptop.

[eladb]

@jogold what do you think? Should we add some support for fixing up permissions during asset staging?

[jogold]

@jogold what do you think? Should we add some support for fixing up permissions during asset staging?

mmmh... only for assets that are going to be used for Lambda then? Not sure we should play with file permissions for Docker assets...

[RomainMuller]

Alright so it turns out this is likely caused by how the @jsii/kernel unpacks the NPM libraries at run-time. When using npm directly (e.g: in a TypeScript app), the problem does not happen, because npm will reset permissions of extracted files (I have yet to establish if those are what is encoded in the tarball, or something else)... But the @jsii/kernel does not do that, and the umask gets applied when we untar (since we do not delegate this to npm).

I reckon the fix is basically to have the @jsii/kernel match npm's permission setting behavior. I'm going to look into this.

[ilkomiliev]

Hi, which CDK Version will integrate the fix, we desperately waiting for this?

Thanks!

[RomainMuller]

Since this is related to the @jsii/kernel, you might already be able to fix yourself by upgrading your dependencies (making sure the jsii library in your install is >= 1.8.0. The upcoming release of CDK will generate dependency requirements on this particular version.