[custom-resources] is being re-deployed on every deployment

[moltar]

Not sure if this is actually a bug, or a feature, but when I am using custom resources, notably AwsCustomResource, it seems to redeploy every time I am doing a deploy. Which takes a few minutes each time.

Reproduction Steps

Use AwsCustomResource and supply onCreate with some basic call.

Error Log

No errors.

Environment

• CLI Version: 1.55.0
• Framework Version: 1.55.0
• Node.js Version: v12.18.2
• OS: macos
• Language (Version): TS

Other

N/A

---

This is 🐛 Bug Report

[eladb]

@jogold any insights?

[jogold]

@moltar this should indeed not happen if you only specify a onCreate. Can you share some code that reproduces the issue?

[moltar]

I'm supplying onUpdate too, sorry for misleading original issue.

[jogold]

I'm supplying onUpdate too, sorry for misleading original issue.

And some properties of your resource change on every deploy?

The long wait time is likely due to the installation of the latest AWS SDK. Will be fixed together with #9289.

[moltar]

Here's my custom resource:

export interface DatabaseMigrationCustomResourceProps {
  vpc: IVpc
  databaseClusterSecret: ISecret
}

export class DatabaseMigrationCustomResource extends AwsCustomResource {
  constructor(
    scope: Construct,
    { vpc, databaseClusterSecret }: DatabaseMigrationCustomResourceProps,
  ) {
    const { name } = DatabaseMigrationCustomResource

    const taskRole = new iam.Role(scope, `${name}TaskRole`, {
      assumedBy: new iam.ServicePrincipal(ServicePrincipals.ECS_TASKS),
      description: 'Role for database migration Fargate Task to migrate the database.',
      inlinePolicies: {
        getSecretValue: new iam.PolicyDocument({
          statements: [
            new iam.PolicyStatement({
              actions: ['secretsmanager:GetSecretValue'],
              resources: [databaseClusterSecret.secretArn],
            }),
          ],
        }),
      },
    })

    const task = new ecs.FargateTaskDefinition(scope, `${name}Task`, {
      taskRole,
      memoryLimitMiB: 8192,
      cpu: 1024,
    })

    task.addContainer(`${name}TaskContainer`, {
      image: ecs.ContainerImage.fromAsset('./'),
      command: ['src/bin/migrate'],
      environment: {
        [DATABASE_URL]: databaseClusterSecret.secretArn,
      },
      logging: new ecs.AwsLogDriver({
        streamPrefix: `${name}TaskContainer`,
      }),
    })

    const ecsCluster = new ecs.Cluster(scope, `${name}TaskCluster`, {
      vpc,
    })

    /**
     * Role used by the AwsCustomResource to apply to the Lambda function that executes
     * on the events.
     */
    const role = new iam.Role(scope, `${name}TaskLambdaRole`, {
      assumedBy: new iam.ServicePrincipal(ServicePrincipals.LAMBDA),
      description:
        'Role used by the AwsCustomResource to apply to the Lambda function that executes on the events.',
      managedPolicies: [
        iam.ManagedPolicy.fromAwsManagedPolicyName(ManagedPolicies.AWS_LAMBDA_BASIC_EXECUTION_ROLE),
        iam.ManagedPolicy.fromAwsManagedPolicyName(
          ManagedPolicies.AWS_LAMBDA_VPC_ACCESS_EXECUTION_ROLE,
        ),
      ],
      inlinePolicies: {
        runTask: new iam.PolicyDocument({
          statements: [
            new iam.PolicyStatement({
              actions: ['iam:PassRole'],
              resources: [ensure(task.executionRole).roleArn, taskRole.roleArn],
            }),

            new iam.PolicyStatement({
              actions: ['ecs:RunTask'],
              resources: [task.taskDefinitionArn],
              // further limits access to a specific cluster only
              conditions: {
                ArnEquals: {
                  'ecs:cluster': ecsCluster.clusterArn,
                },
              },
            }),
          ],
        }),
      },
    })

    const parameters: ECS.Types.RunTaskRequest = {
      launchType: 'FARGATE',
      count: 1,
      cluster: ecsCluster.clusterArn,
      taskDefinition: task.taskDefinitionArn,
      networkConfiguration: {
        awsvpcConfiguration: {
          assignPublicIp: 'ENABLED',
          subnets: vpc.selectSubnets().subnetIds,
        },
      },
    }

    const awsSdkCall: AwsSdkCall = {
      service: 'ECS',
      action: 'runTask',
      physicalResourceId: {
        id: task.taskDefinitionArn,
      },
      parameters,
    }

    super(scope, name, {
      resourceType: `Custom::${name}`,
      policy: AwsCustomResourcePolicy.fromSdkCalls({
        resources: AwsCustomResourcePolicy.ANY_RESOURCE,
      }),
      role,
      onCreate: awsSdkCall,
      onUpdate: awsSdkCall,
    })
  }
}

Here's what I get in the events:

[moltar]

I am thinking maybe I shouldn't be piling all of that into the AwsCustomResource sub class but create a nested stack instead?

[moltar]

Switched to:

export class DatabaseMigrationCustomResource extends Construct {

but it made no difference.

A simple one character change to the app code will create 4 CF changes, and the deployment takes ~ 6 minutes to complete.

[jogold]

@moltar you can now disable the latest SDK installation, it will speed up things.

[moltar]

@moltar you can now disable the latest SDK installation, it will speed up things.

Thank you for looking into this.

According to the log it seems to only add a max of one minute. But I'm seeing a tiny change take 6 minutes to deploy. That's without the upload and so on. Just CF changes.

[jogold]

I think that's because your have multiple custom resources. Let's discuss this again when #9515 is released.

[kisharma24]

do you have any logic where this migration will happen automatically with every pipeline deployment? (maybe using timestamp or something)