codepipeline: CloudFormationCreateUpdateStackAction for cross-account deployment

[soleyman-devops]

Describe the bug

Resource handler returned message: "The role with name CrossAccountDevDeploy cannot be found"

I'm creating cross account cicd, i've deployed the CrossAccountDevDeploy role in my Dev Account, and created a trusted entity in my CICD account to assume this role, and passed the CICD Account ID in my Dev Role.

However when I use the role in my dev deploy stage with codepipeline it cannot find the CrossAccountDevDeploy

CICD Pipeline

export class CicdPipeline extends cdk.Stack {
    constructor(scope: Construct, id: string, props?: cdk.StackProps) {
      super(scope, id, props);
  
      // Pipeline IAM
      const pipelineRole = new iam.Role(this, 'CustomPipelineRole', {
        assumedBy: new iam.ServicePrincipal('codepipeline.amazonaws.com'),
      })
  
      // Pipeline to Secrets Manager Github Token
      pipelineRole.addToPolicy(new iam.PolicyStatement({
        actions: ['secretsmanager:GetSecretValue', 'secretsmanager:DescribeSecret', 'sts:AssumeRole'],
        resources: ['arn:aws:secretsmanager:eu-west-1:xxxxxxxxxxxx:secret:xxxxxxxxxxxx', 'arn:aws:iam::xxxxxxxxxxxx:role/CrossAccountDevDeploy'],
      }))
 
  
      // Output Artifacts
      const sourceOutput = new codepipeline.Artifact();
      const cdkOutputs = new codepipeline.Artifact('CDKOutputs')
  
      // CDK Build Stage
      const cdkBuild = new codebuild.Project(this, 'CDKBuild', {
        buildSpec: codebuild.BuildSpec.fromObject({
          version: '0.2',
          phases: {
            install: {
              commands: ['npm install -g aws-cdk', 'npm install']
            },
            build: {
              commands: ['cdk synth']
            },
          },
          artifacts: {
            'base-directory': 'cdk.out',
            files: ['*']
          }
        }),
  
        // Runtime env for CodeBuild
        environment: {
          buildImage: codebuild.LinuxBuildImage.STANDARD_5_0
        }
      })
  
      // Pipeline itself
      new codepipeline.Pipeline(this, "Pipeline", {
        pipelineName: 'Foundational-Pipeline',
        role: pipelineRole,
        stages: [
          {
            stageName: 'Source',
            actions: [
              new codepipelineActions.GitHubSourceAction({
                actionName: 'Github',
                repo: 'foundation',
                oauthToken: cdk.SecretValue.secretsManager('xxxxxxxxxxxx'),
                output: sourceOutput,
                owner: 'pipeline',
                trigger: codepipelineActions.GitHubTrigger.WEBHOOK
              })
            ]
          },
          // Build CDK into CloudFormation
          {
            stageName: 'Build',
            actions: [
              new codepipelineActions.CodeBuildAction({
                actionName: 'CDK_Build',
                project: cdkBuild,
                input: sourceOutput,
                outputs: [new codepipeline.Artifact('CDKOutputs')]
              })
            ]
  
          },
          // Deploy to Dev Stage
          {
            stageName: 'DeployDev',
            actions: [
              new codepipelineActions.CloudFormationCreateUpdateStackAction({
                actionName: 'Deploy',
                stackName: 'DevFoundationalNetworking',
                templatePath: cdkOutputs.atPath('NetworkingDev.template.json') ,
                adminPermissions: false,
                role: iam.Role.fromRoleArn(this, 'CrossAccountRole', 'arn:aws:iam::xxxxxxxxxxxx:role/CrossAccountDevDeploy')
              })
            ]
          }

dev cross account role

          export class DevAccount extends cdk.Stack {
            constructor(scope: Construct, id: string, props?: cdk.StackProps) {
              super(scope, id, props);
          
              const cicdAccountId = 'xxxxxxxxxx'
          
              const devDeployRole = new iam.Role(this, 'DevDeployRole', {
                assumedBy: new iam.AccountPrincipal(cicdAccountId),
                roleName: 'CrossAccountDevDeploy',
                description: 'Allows CICD account to deploy CloudFormation in dev'
              })
          
              devDeployRole.addToPolicy(new iam.PolicyStatement ({
                actions: ['*'],
                resources: ['*']
              }))
            }
          }
          
``` ### 

### Expected Behavior

To assume the CrossAccountDevDeploy role in Dev Account from CICD Account and enable the pipeline to deploy

### Current Behavior

The above bug Failure when deploying the update dev deploy stage. 

### Reproduction Steps

I deploy the pipeline using 'cdk deploy --profile xxx' it connects to cicd account and starts the deployment but errors . 

### Possible Solution

_No response_

### Additional Information/Context

Ive removed any sensestive data from the code.

### CDK CLI Version

2.96.2

### Framework Version

_No response_

### Node.js Version

v20.6.1

### OS

Mac

### Language

TypeScript

### Language Version

_No response_

### Other information

ive dug through the internet and they all say make sure the ARN and account ID are correct in dev account which they are. theres something not right in my cicd account reading the CrossAccountDevDeploy

[peterwoodworth]

Could you more clearly describe how the DevAccount stack is getting deployed? When you bootstrapped did you include the --trust or --trust-for-lookup options when bootstrapping to allow for lookups cross account?

[soleyman-devops]

@peterwoodworth DevAccount stack is deployed and it was CDK bootstrapped, however I did not add the --trust or --trust-for-lookup options. I will do this now and get back to you

[soleyman-devops]

@peterwoodworth - still having the same error, after adding the --trust to the bootstrap for the DevAccount.

cdk bootstrap --cloudformation-execution-policies arn:aws:iam::aws:policy/AdministratorAccess --trust xxxxxxx aws://xxxxxxxxx/eu-west-1

[pahud]

This could be a little bit confusing but I think the general idea is to clarify the 3 roles

1. pipeline role - the main role the pipeline run with
2. action role - the role that pipeline role assumes when executing this action. In this case, CloudFormationCreateUpdateStackAction and is defailed as role property.
3. cloudformation deploy role - the action role would use this role to deploy cloudformation which is defined as deploymentRole.

I guess you should specify CrossAccountDevDeploy as deployment role instead.

Now, if you look at the synthesized template as below, your CrossAccountDevDeploy should be at 1 rather than 2.

Let me know if it works for you. It's a little bit confusing to be honest.

[github-actions]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

[soleyman-devops]

Hey @pahud @peterwoodworth I've gone through the code and have done the required steps, ie validated permissions from my dev deployment role allows my pipeline role in cicd source account to be assumed.

So pipeline role has permissions to assume dev deployment role from target account and vice versa.

However, I am still getting the same error.

Ive bootstrapped too as a extra step but still no luck.

export class AwsCicdStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // Pipeline IAM
    const pipelineRole = new iam.Role(this, 'CodePipelineRole', {
      roleName: 'FoundationCodePipelineRole',
      assumedBy: new iam.ServicePrincipal('codepipeline.amazonaws.com'),
    })

    // // Pipeline Assume Dev Role
    pipelineRole.addToPolicy(new iam.PolicyStatement({
      actions: ['sts:AssumeRole'],
      resources: [ 
        'arn:aws:iam::devaccountid:role/Dev-Deployment-Role'
      ],
    }))

    // Output Artifacts
    const sourceOutput = new codepipeline.Artifact('SourceArtifact');
    const cdkOutputs = new codepipeline.Artifact('CDKOutputs')

    // CDK Build Stage
    const cdkBuild = new codebuild.PipelineProject(this, 'CDKBuild', {
      buildSpec: codebuild.BuildSpec.fromObject({
        version: '0.2',
        phases: {
          install: {
            commands: ['npm install -g aws-cdk', 'npm install']
          },
          build: {
            commands: ['npm run cdk synth']
          },
        },
        artifacts: {
          'base-directory': 'cdk.out',
          files: [`*.template.json`],
        }
      }),
      // Runtime env for CodeBuild
      environment: {
        buildImage: codebuild.LinuxBuildImage.STANDARD_5_0
      },
      // encryptionKey: key
    })
const pipeline = new codepipeline.Pipeline(this, "Pipeline", {
      pipelineName: 'Foundational-Pipeline',
      crossAccountKeys: true,
      role: pipelineRole,
      stages: [
        {
          stageName: 'Source',
          actions: [
            new codepipelineActions.GitHubSourceAction({
              actionName: 'Github',
              repo: 'ops-aws-foundation',
              branch: 'main',
              oauthToken: cdk.SecretValue.secretsManager('xxxx'),
              output: sourceOutput,
              owner: 'xxxx',
              trigger: codepipelineActions.GitHubTrigger.WEBHOOK
            })
          ]
        },
        // Build CDK into CloudFormation
        {
          stageName: 'Build',
          actions: [
            new codepipelineActions.CodeBuildAction({
              actionName: 'CDK_Build',
              project: cdkBuild,
              input: sourceOutput,
              outputs: [new codepipeline.Artifact('CDKOutputs')],
              runOrder: 1
            })
          ]
        },
        {
          stageName: 'DeployDev',
          actions: [
            new codepipelineActions.CloudFormationCreateUpdateStackAction({
              actionName: 'DeployNetworkingStack',
              stackName: 'FoundationalNetworking',
              templatePath: cdkOutputs.atPath('AwsFoundationStack.template.json'),
              adminPermissions: true,
              deploymentRole: iam.Role.fromRoleArn(this, 'role', 'arn:aws:iam::devaccountid:role/Dev-Deployment-Role')
          })
          ]
        },
      ]
    });

    pipeline.addToRolePolicy(new iam.PolicyStatement({
      actions: ['sts:AssumeRole'],
      resources: [
        `arn:aws:iam::devaccountid:role/Dev-Deployment-Role`
      ]
    }))
  }
}

`

Any help would be appreciated

[pahud]

I will try to provide a working sample but before that, can you share the synthesized cloudformation template of the "Actions" as the screenshot above?

[pahud]

Hi @soleyman-devops I will try to write a small sample for this. Hopefully to clarify some details.

[pahud]

Please check the full sample below:

Assuming the pipeline account 111111111111 and the target deployment account 222222222222. Both the pipeline and deployment stack are in us-east-1.

1. Make sure you cdk bootstrap 222222222222 with --trust for 111111111111

# in account 222222222222
cdk bootstrap aws://222222222222/us-east-1 --trust 111111111111 --cloudformation-execution-policies arn:aws:iam::aws:policy/AdministratorAccess

check the iam policies of cdk-hnb659fds-deploy-role-222222222222-us-east-1 iam role, make sure the trust policy is correctly configured as:

{
    "Version": "2008-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::222222222222:root"
            },
            "Action": "sts:AssumeRole"
        },
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::111111111111:root"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

2. Now let's create the following CDK stack for the pipeline

export class Demo extends DemoStack {
	constructor(scope: Construct, id: string, props: StackProps) {
		super(scope, id, props);

		const sourceOutput = new codepipeline.Artifact();

		new codepipeline.Pipeline(this, 'pipeline', {
			stages: [
				// source stage
				{
					stageName: 'Source',
					actions: [
						new codepipeline_actions.CodeStarConnectionsSourceAction({
							actionName: 'GithubSource',
							connectionArn,
							owner: 'pahud',
							repo: 'test-pipeline',
							branch: 'main',
							output: sourceOutput,
						})
					]
				},
				// deploy stage
				{
					stageName: 'cfn-deploy',
					actions: [
						new codepipeline_actions.CloudFormationCreateUpdateStackAction({
							actionName: 'CloudFormationCreateUpdate',
							stackName: 'MyStackName',
							adminPermissions: true,
							templatePath: sourceOutput.atPath('template.yaml'),
							// the pipeline role will assume this role to run this deploy action, which passes
							// the `deploymentRole` to cloudformation to assume.
							role: iam.Role.fromRoleArn(this, 'actionRole', 
								'arn:aws:iam::222222222222:role/cdk-hnb659fds-deploy-role-222222222222-us-east-1'),
							// cloudformation will deploy the stack using this role
							deploymentRole: iam.Role.fromRoleArn(this, 'deployRole', 
							'arn:aws:iam::222222222222:role/cdk-hnb659fds-cfn-exec-role-222222222222-us-east-1'),
						}),
					]
				}

			]
		});
	
	}	
}

3. Now cdk diff or cdk synth, check the json template in the cdk.out. You should see the Actions like

   "Actions": [
       {
        "ActionTypeId": {
         "Category": "Deploy",
         "Owner": "AWS",
         "Provider": "CloudFormation",
         "Version": "1"
        },
        "Configuration": {
         "StackName": "MyStackName",
         "Capabilities": "CAPABILITY_NAMED_IAM",
         "RoleArn": "arn:aws:iam::222222222222:role/cdk-hnb659fds-cfn-exec-role-222222222222-us-east-1",
         "ActionMode": "CREATE_UPDATE",
         "TemplatePath": "Artifact_Source_GithubSource::template.yaml"
        },
        "InputArtifacts": [
         {
          "Name": "Artifact_Source_GithubSource"
         }
        ],
        "Name": "CloudFormationCreateUpdate",
        "RoleArn": "arn:aws:iam::222222222222:role/cdk-hnb659fds-deploy-role-222222222222-us-east-1",
        "RunOrder": 1
       }
      ],
      "Name": "cfn-deploy"
     }
    ]

This means the pipeline role will assume the "action role" to run the cloudformation action by passing the "deploymentRole" to the cloudformation service. The "action role" and "deploymentRole" have to be both in the target account.

In this case:

• the pipeline role would be auto generated by CDK
• action role is cdk-hnb659fds-deploy-role-222222222222-us-east-1, the pipeline role will assume this role
• deployment role is cdk-hnb659fds-cfn-exec-role-222222222222-us-east-1, the action role will pass this role to cloudformation service.

4. On deploy completed, the pipeline should be triggered and you should see this from the codepipeline console:

5. Now, check the cloudformation console in account 222222222222, the stack should be successfully deployed.

Let me know if it works for you.

[soleyman-devops]

Hi Paul, thanks for taking time out to send me the example. I followed the steps, but getting this action role cannot be found.

Everything matches what you shared above until deployment of the CodePipeline stack.
I double checked and its definitely present in the target account. Seems strange that it cannot find it even after bootstrapping.

[pahud]

@soleyman-devops Interesting. If it says this role cannot be found then it technically does not exist.

Are you able to get-role with AWS CLI like this in your target account?

 aws iam get-role --role-name cdk-hnb659fds-deploy-role-<your_target_account_id>-us-east-1

Are you able to reproduce from my provided code snippets above and see if it works for you?

[github-actions]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.