aws-lambda: version resource name changing from v1 to v2

[pcolmer]

Describe the bug

Code that deploys a Lambda is generating a resource that references the current version. The name of that resource is changing when we migrate from CDK v1 to v2, resulting in CloudFormation wanting to destroy the existing resources but it can't because there are other stacks in the app that are dependent on those resources.

The only way I seem to be able to deploy the app using v2 CDK is to destroy it completely first which is undesirable.

Expected Behavior

I should be able to deploy/upgrade my app to v2 without problems and certainly without needing to destroy the entire deployment.

Current Behavior

Running diff results in the following:

Resources
[-] AWS::Lambda::Version cloudfrontaddsecurityheadersCurrentVersion16169602d5819224674252586b3d48dd18654174 destroy
[-] AWS::Lambda::Version addindextodirectoryurlsCurrentVersionD6A21E4798a50b14deec652bcdbf1dbe6d635381 destroy
[+] AWS::Lambda::Version cloudfront-add-security-headers/CurrentVersion cloudfrontaddsecurityheadersCurrentVersion1616960298ca6e043ddae0d5ff51b3ecf3edb258
[+] AWS::Lambda::Version add-index-to-directory-urls/CurrentVersion addindextodirectoryurlsCurrentVersionD6A21E472e930828858ae2ecd8cfb4658a1927a5

Reproduction Steps

• Using CDK v1, create hello-cdk app.
• Modify hello_cdk>hello_cdk_stack.py thus:

from aws_cdk import core as cdk
from aws_cdk import aws_lambda as lambda_

CLOUDFRONT_ADD_SECURITY_HEADERS_NAME = "pjc-test-headers"

class HelloCdkStack(cdk.Stack):

    def __init__(self, scope: cdk.Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        lambda_security_header = lambda_.Function(
            self,
            CLOUDFRONT_ADD_SECURITY_HEADERS_NAME,
            runtime=lambda_.Runtime.NODEJS_12_X,
            code=lambda_.Code.from_asset("lambda/add-security-headers"),
            handler="add-security-headers.handler",
            function_name=CLOUDFRONT_ADD_SECURITY_HEADERS_NAME
        )
        self.add_security_headers_arn = lambda_security_header.current_version.edge_arn

• cdk1 deploy
• Look at the resources generated by CloudFormation and note that the Lambda version resource has a name like pjctestheadersCurrentVersion6A41A374a584fa336027d0271a4c8a7f1566b8c0
• Modify the code to make it v2 compatible.
• cdk diff

This will produce output like this:

[-] AWS::Lambda::Version pjctestheadersCurrentVersion6A41A374a584fa336027d0271a4c8a7f1566b8c0 destroy
[+] AWS::Lambda::Version pjc-test-headers/CurrentVersion pjctestheadersCurrentVersion6A41A374f92b3f06e16da8e8e102c48b89c67cef

Note that setting @aws-cdk/core:stackRelativeExports to false does not fix the issue.

Possible Solution

None known.

Additional Information/Context

No response

CDK CLI Version

1.159.0 and 2.27.0

Framework Version

No response

Node.js Version

16.3.0

OS

Ubuntu 20.04.0

Language

Python

Language Version

No response

Other information

No response

[peterwoodworth]

I don't think we changed anything about our hashing or name generation between versions, so it's strange to me that you're running into this

We're in maintenance mode for v1, so since this isn't a major issue we won't be addressing it. You can work around this by overriding the logical ids of resources which are having their names changed.

[pcolmer]

@peterwoodworth

I don't think we changed anything about our hashing or name generation between versions, so it's strange to me that you're running into this

Which is why I've reported it!

The documentation at https://docs.aws.amazon.com/cdk/v2/guide/migrating-v2.html says:

If your upgraded app ends up undeployable after the two-stage upgrade, please report the issue.

I am unable to re-deploy my app after upgrading from CDK v1 to CDK v2, hence my reporting this issue. At the moment, without making any code modifications, my only recourse is to destroy the stack and then do a new deploy of the stack, which results in a complete loss of the infrastructure during that process.

I have, as required by the guidelines, provided a small (2 lines of code!) 100% reproducible example of the problem I'm having.

You can work around this by overriding the logical ids of resources which are having their names changed.

So are you proposing that I hack my code - for each deployment of the stack - to override the "wrong" logical ID with the logical ID that has already been deployed? Without knowing how many instances I've deployed ... ?

Or are you proposing that there is a way for me to (a) see if the stack has already been deployed, (b) retrieve the logical ID of the resource if it has been deployed, (c) continue with the rest of the code and then (d) override the "wrong" ID with the old one? If so, can you please provide more guidance/example code?

If I truly have to destroy existing deployments in order to upgrade to CDK v2 then perhaps AWS CDK is not the tool for me after all.

[peterwoodworth]

I have, as required by the guidelines, provided a small (2 lines of code!) 100% reproducible example of the problem I'm having.

The issue here is that I'm not able to reproduce the behavior you're seeing with the code you've provided.

So are you proposing that I hack my code - for each deployment of the stack - to override the "wrong" logical ID with the logical ID that has already been deployed? Without knowing how many instances I've deployed ... ?

I merely proposed a method which can work 🙂 I know it's far from ideal, but I figured it was worth mentioning since I knew it off the top of my head

Are you able to reproduce this issue following your steps in a completely fresh project?

[pcolmer]

Are you able to reproduce this issue following your steps in a completely fresh project?

Yes - I wrote the issue as I built the fresh project. I literally did what I wrote. I deployed the original code using CDK v1, followed the instructions to upgrade the code/stack to CDK v2 then attempted to deploy using v2.

I should point out that if you deploy JUST using CDK v2, you won't see the problem. This only seems to be apparent when upgrading an existing stack that was deployed with v1.

The issue here is that I'm not able to reproduce the behavior you're seeing with the code you've provided.

That is worrying. I will go through it all again to make sure I haven't missed anything out or mistyped anything at my end.

[pcolmer]

@peterwoodworth I've put together a git repository with two branches that definitely repros the issue: https://github.com/pcolmer/aws-cdk-20650

The main branch is the CDK v1 version of the app. After deploying that, switch to the CDK v2 branch and follow the steps in the README to upgrade the environment to v2 and then cdk diff.

If you still cannot reproduce it after that, then I don't know what to say. I'm open to running any extra commands or diagnostics if it helps unpack why the resource ID is getting changed.

[peterwoodworth]

I see why you're running into this @pcolmer -

This is where we calculate the logical ID - we actually override it ourselves -

aws-cdk/packages/@aws-cdk/aws-lambda/lib/function.ts

Lines 421 to 433 in ba91ca4

	// override the version's logical ID with a lazy string which includes the
	// hash of the function itself, so a new version resource is created when
	// the function configuration changes.
	const cfn = this._currentVersion.node.defaultChild as CfnResource;
	const originalLogicalId = this.stack.resolve(cfn.logicalId) as string;
	cfn.overrideLogicalId(Lazy.uncachedString({
	produce: () => {
	const hash = calculateFunctionHash(this);
	const logicalId = trimFromStart(originalLogicalId, 255 - 32);
	return `${logicalId}${hash}`;
	},
	}));

We override the last 32 characters based of the hash of the lambda function. The difference between your versions is only on the last 32 characters. I've also managed to reproduce this myself on my own TS library and also with your repo.

The difference between v1 and v2 I think is due to the fact that the hash is based on the lambda function and its properties. There is a difference in how the S3key prop is built between the versions - but that's handled deep within our asset code

aws-cdk/packages/@aws-cdk/core/lib/stack-synthesizers/_asset-manifest-builder.ts

Line 18 in ba91ca4

public addFileAssetDefault(

Without looking into it more I'm not super sure why there's a difference between versions. But this is what I've found so far

[pcolmer]

@peterwoodworth Thank you for continuing to investigate and for sharing your findings so far. I am so relieved that you have been able to reproduce the problem now!

The difference between your versions is only on the last 32 characters.

Ah! Upon re-reading the output from cdk diff, I can see what you mean! I was seeing this:

[-] AWS::Lambda::Version pjctestheadersCurrentVersion6A41A3741e769839a4a4eca15363055ff023227a destroy
[+] AWS::Lambda::Version pjc-test-headers/CurrentVersion pjctestheadersCurrentVersion6A41A374f92b3f06e16da8e8e102c48b89c67cef

and thinking that CDK was destroying pjctestheadersCurrentVersion6A41A3741e769839a4a4eca15363055ff023227a and wanting to create a new resouce called pjc-test-headers/CurrentVersion.

I had completely misunderstood that output and now see that it is wanting to create pjctestheadersCurrentVersion6A41A374f92b3f06e16da8e8e102c48b89c67cef

Will you or someone on the CDK team be able to look into this further or is it something I've got to "live with" and just hack my code to get past the issue?

[indrora]

@peterwoodworth @kaizencc would a feature flag for those coming from old versions help?

or do we consider this a "This is a price you pay for updating"?

[Rabadash8820]

@indrora This is definitely not a price that anyone is expecting to pay when they update, considering that the migration docs say:

Changes to logical IDs (causing replacement of resources) are not expected.

I've been running into this issue myself. I'm still confused why the logical IDs are changing at all if neither the function configuration nor the function code have changed. @peterwoodworth do you have any other insight into that?

[Mickael-van-der-Beek]

This is a serious issue for us as well and prevents us from upgrading to v2.

SecurityGroups get randomly removed and added back again with different names and various cross-stack references get renamed as well creating dead-locks prevents deployment of stacks.

I would consider not being able to upgrade to v2 in scope.

[jonife]

This is a known issue when migrating from CDK V1 to V2 and the workaround proposed here, seems to be the best options necessary to ensure that this issue is not a problem anymore.