Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RFC 66: Self Managed Stack Set Support #357

Closed
wants to merge 8 commits into from
Closed
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
329 changes: 329 additions & 0 deletions text/66-self-managed-stack-sets.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,329 @@
---
linsona marked this conversation as resolved.
Show resolved Hide resolved
rfc pr: [#347](https://github.com/aws/aws-cdk-rfcs/pull/347)
tracking issue: https://github.com/aws/aws-cdk-rfcs/issues/66
---

# CDK Self Managed Stack Set Support
linsona marked this conversation as resolved.
Show resolved Hide resolved

As a CDK user, I would like to define stack set infrastructure through native CDK constructs,
and create/update self-managed stack sets through CDK deploy command.

* **Original Author(s):** @linsona
* **Tracking Issue**: #66
* **API Bar Raiser**: @skinny85

## Working Backwards

### CHANGELOG

* feat(assembly): add stack set artifact type and schema
* feat(core): add `StackSet` construct
* feat(cli): CLI commands like `deploy` now support stack sets
* feat(bootstrap): add IAM Roles for performing CDK stack set deployments

### README

The `cdk.StackSet` construct in CDK defines AWS resources in a stack set and stack set
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please replace "stack sets" (or any variation thereof) with "StackSets" (one word) across this doc. I believe this is the how it is used by the service documentation.

configuration/deployment preferences.
This is different from the `cdk.CfnStackSet` construct, which requires a CloudFormation stack that then deploys a CloudFormation stack set.
The `cdk.StackSet` construct creates the stack set directly if it does not exist,
or updates a stack set and existing stack set instances while monitoring the stack set operation.
Comment on lines +28 to +30
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this better than what CfnStackSet does?


Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add some information about what StackSets are (from the official CFN docs) and a reference to the official docs entry point

#### Prerequisites
linsona marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor:

Suggested change
#### Prerequisites
#### Bootstrapping


To enable the use of self managed stack sets two roles must exist:

* **Stack Set Administration Role:** Role in the parent account, which is used by CloudFormation to assume the stack set execution role in the child account.
* **Stack Set Execution Role:** Role in the child accounts, which allows the stack set administration role in parent account to assume,
and perform the stack set instance deployment.
Comment on lines +36 to +38
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These roles sounds very much like the ones we use for CDK pipelines. Can't we reuse them?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. I'm thinking the same.

If all we need is some slight adjustment to the existing role, then we don't need to change the cdk bootstrap command.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, so after discussing with Adam, we decided to go the route of introducing new roles:

  • The Stack Set Execution Role names should be identical in all accounts, which is a requirement for how stack sets operate. The current roles in the bootstrap template are suffixed with specific account ids
  • The Stack Set Execution Role also must trust the Stack Set Administration Role, to assume and provision the stack set instance.
    • In the current bootstrap, the trust flag would give any IAM entity in the account permissions to assume the execution role, which may have a large scope of permission.
    • This is different from trust given to the DeploymentActionRole, where it only has permissions to to create stack/changeset, and the actual role with large scope of permissions is CloudFormationExecutionRole which is not assumable to by the trusted account


There are a few methods to creating the stack set roles:

1. CDK Bootstrap with default stack set roles
linsona marked this conversation as resolved.
Show resolved Hide resolved
* Bootstrap parent account
* `cdk bootstrap aws://11111111111/us-east-1 --stack-set`
* The region here is where the stack set will be created in the parent account.
* Bootstrap child account
* `cdk bootstrap aws://22222222222/us-east-1
--stack-set
--trust 11111111111
--cloudformation-execution-policies arn:aws:iam::aws:policy/AdministratorAccess`
* The child account **must** be bootstrapped to the same region where the parent account is bootstrapped.
Although bootstrapping must be in the same region,
stack set instances can still be provisioned in any region of the same AWS partion in the child account.
* Specifying custom qualifier:
* `cdk bootstrap aws://11111111111/us-east-1 --qualifier xyz`
* `cdk bootstrap aws://22222222222/us-east-1
--trust 11111111111
--cloudformation-execution-policies arn:aws:iam::aws:policy/AdministratorAccess
--qualifier xyz`
* Qualifier must match in both the parent and child accounts.
2. Manually create roles
* AWS Guide: [Grant self-managed permissions](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/stacksets-prereqs-self-managed.html)
* The AWS documentation will guide you through creating the Stack Set Administration role for the parent account,
and the Stack Set Execution roles for the child accounts.
* The administration role and execution role name should be specified as part of the StackSet properties.

#### Usage
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The main value of a stack set is to deploy a template to different accounts (and regions).

The usage should talk about this primary use case - how to specify the list of target accounts and regions for this stack set.

And also cover, how to configure parameters that will have different values in the target accounts/regions.


To define a stack set, extend the core construct `StackSet`.
Then define AWS resources in the same way as defining resources for stacks.
Compared to the `Stack` construct, the `StackSet` core construct has different properties specific to StackSets.
The properties define the configuration of the stack set and the preferences used for update operations.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are there any limitations on StackSets comparing to normal stacks? If there are, this is where they should be listed.

**Example:**

```ts
linsona marked this conversation as resolved.
Show resolved Hide resolved
import * as cdk from '@aws-cdk/core';
import * as iam from '@aws-cdk/aws-iam';

class PocStackSet extends cdk.StackSet {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Our convention is not to use namespaced imports for core types:

Suggested change
class PocStackSet extends cdk.StackSet {
class MyStackSet extends StackSet {

constructor(scope: cdk.Construct, id: string, props?: cdk.StackSetProps) {
super(scope, id, props);

new iam.Role(this, 'ExampleRole', {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would be nice to include a bit more involved example (i.e. add a bucket or something)

assumedBy: new iam.ServicePrincipal('s3.amazonaws.com')
});
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change to 2 space indentation please

}
}
linsona marked this conversation as resolved.
Show resolved Hide resolved

// Using default cdk bootstrap stack set roles and default qualifier
new PocStackSet(app, 'poc-stack-set-1', {
faultTolerancePercentage: 10,
maxConcurrentPercentage: 10,
regionConcurrencyType: cdk.StackSetRegionConcurrency.PARALLEL,
//...
});

// Using default cdk bootstrap stack set roles with custom qualifier
new PocStackSet(app, 'poc-stack-set-2', {
qualifier: 'xyz',
faultTolerancePercentage: 10,
maxConcurrentPercentage: 10,
regionConcurrencyType: cdk.StackSetRegionConcurrency.PARALLEL,
//...
});

// Using customized stack set roles
Comment on lines +98 to +107
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To maintain a good flow for the document, I would put all the information about custom bootstrapping, roles and qualifiers to a separate section at the end "Customizing StackSet Bootstrapping".

new PocStackSet(app, 'poc-stack-set-3', {
administrationRoleName: 'AWSCloudFormationStackSetAdministrationRole',
executionRoleName: 'AWSCloudFormationStackSetExecutionRole',
Comment on lines +109 to +110
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't this assuming that the execution role will be the same in all accounts? Should it be more flexible than that?

faultTolerancePercentage: 10,
maxConcurrentPercentage: 10,
regionConcurrencyType: cdk.StackSetRegionConcurrency.PARALLEL,
//...
});
```

**Deploy Stack Set:**

To deploy the stack set, the CLI command remains the same: `cdk deploy poc-stack-set`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I understand this is mentioned just after, but when I read this first it wasn't clear to me that when I first deploy a stack set, it won't deploy any instances. Maybe merge the two sections about "deploy stackSets" and "creating StackSet instances" into a single section about deployment.

If the stack set **does not exist**, then CDK will create the stack set directly in CloudFormation.
If the stack set **does exist**,
then the stack set itself will be updated as well as all **existing** stack set instances.

**Creating Stack Set Instances:**

Before creating stack set instances, child accounts should be bootstrapped (or custom stack set roles created) and the initial stack set deployed.
Stack set instances can be added to the stack set by console/cli/api by specifying the child account id and region.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How exactly? Why not via the cdk?

Future stack set deployments though CDK will update the stack set itself, and all existing stack set instances.

## FAQ

### What are we launching today?

The core construct StackSet, which will enable users to define stack sets directly from CDK.
A new Cloud Assembly artifact type for stack sets. CLI changes to enable deploying stack sets through CDK.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The CDK's cloud assembly is a public API. If we're making changes there (to the schema), the working backwards section should include the API changes there.


### Why should I use this feature?

This core constuct directly creates a CloudFormation stack set vs. provisioning a single stack that deploys a stack set.
The deployment also updates the stack set, updates existing stack set instances, and monitors the stack set operation.
Comment on lines +138 to +141
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not answering the question. What is the benefit of this change? "Why should I use it?"


### What is supported?

* Self Manged Stack Sets
* Create, Update, and Update Existing Stack Set Instances directly through CDK

### What is not supported?

* Service Managed Stack Sets
* Stack Set Instance Add/Remove through CDK
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From what I understand, 'remove instance from stack set' is a very important use case for stack sets.
Without this, we will corner users into a situation where a production stack is bound to a stack set and they lose flexibility.

Will users be able to do this outside of the cdk? If so, we should have a section in the README to talk about this and the impact of an 'out of band' action.

* Stack Set Deletion
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What would be the experience of running "cdk destroy"? Maybe we can support cdk destroy for StackSets that do not have instances?

* Stack Set deletion will not be included due complexities with collecting/deleting stack set instances before deleting the stack set itself
* CDK Assets (CDK File Assets/CDK Docker Assets)
* CDK Assets will not be include in the iteration due to complexities around asset permissioning accross accounts and regions

## Internal FAQ

### Why are we doing this?

Today, CDK does not support the creation and updating of CloudFormation stack sets directly.
This solution would also enable deploying to existing stack sets with low effort.

CDK does provide the `cdk.CfnStackSet` construct, which could accomplish similar behavior.
The difference is that the `cdk.CfnStackSet` construct is a resource in a single CloudFormation stack with the resource type `AWS::CloudFormation::StackSet`,
where stack provisions the actual stack set.
This requires a 1:1 mapping of parameters and tags that need to be passed from the stack to the stack set.
Customer can deploy to existing stack sets, but requires some migration steps.
The customer would need to create a new stack,
and import the stack set into the resource `AWS::CloudFormation::StackSet`.
Comment on lines +159 to +170
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm mostly repeating myself here, but this doesn't cover why this approach is better than using CfnStackSet.

We should be talking about this in terms of what we are enabling for users here that didn't previously exist.


### Why should we _not_ do this?

We might not want to do this since CDK does provide the `cdk.CfnStackSet` construct, which could accomplish similar behavior.
The intermdiate stack may not be ideal for some customers, and may require large migration effort for customers who manage many stacksets.

### What changes are required to enable this change?

To enable this feature, we would need to introduce:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about bootstrap stack changes?


* A `StackSet` construct for customers to extend
* A new assembly/artifact type for the Cloud Assembly
* A new stack set synthesizer
linsona marked this conversation as resolved.
Show resolved Hide resolved
* A new stack set synthesizer is required to emit new stack set artifact and properties to the CloudAssembly
* Deployment code that creates/updates stack sets using AWS CloudFormation APIs

linsona marked this conversation as resolved.
Show resolved Hide resolved
### Is this a breaking change?

No.

### What are the drawbacks of this solution?

The handling of CDK assets is more complex than a normal stack CDK app.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds like assets are not going to be supported anyway

Stack set instances are deployed independently from CDK context, and exist in different accounts/regions.
This is dicussed more in the design details.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
This is dicussed more in the design details.
This is discussed more in the design details.


### What alternative solutions did you consider?

An alternative is the usage of `cdk.CfnStackSet`, which is not ideal for customers with existing stack sets since there is significant migration effort.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have to say that this does not sound like a good enough reason not to implement this. Migrating from existing infrastructure (either CFN-based or not) is hard in most cases...

What are other reasons for us to support this?


### What is the high level implementation plan?

* Update default boostrap template to include default stack set roles
* Create `StackSet` core construct
* Create StackSet artifact type in the Cloud Assembly
* Create StackSet Synthesizer
* Update CLI to collect StackSet artifact types in addition to Stacks
* Add deployment code for creating/updating stack sets

linsona marked this conversation as resolved.
Show resolved Hide resolved
### Are there any open issues that need to be addressed later?

* CDK asset handling
* Future support for service managed stack sets
* Future support for delete stack set
* CDK Pipelines

## Appendix

### CDK Bootstrap CLI Changes

To support stack sets,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Repeat from above: can we use/expand the CDK pipelines roles we already have in the bootstrap stack? Intuitively these should suffice since CDK Pipelines is basically doing the same thing as stack sets.

a stack set administration role should be created in the parent account and a stack set execution role should be created in the child accounts.
To make this process easier, default roles have been added to the default bootstrap template. (Bootstrapping steps describe in the README section).

**CLI Changes:**

* Update bootstrap CLI to include new flag `--stack-set`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* Update bootstrap CLI to include new flag `--stack-set`
* Update bootstrap CLI to include new flag `--stackset`

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If possible, it would be nice if we didn't need any of this

* If this flag is present, the bootstrap stack will create the stack set roles.
* The bootstrap CLI will deploy the stack with the parameter `EnableCreateStackSetRoles` set to `true`.
* Update bootstrap CLI to generate administration role arns from the `--trust` argument
* If the flag `--stack-set` and `--trust` argument is present,
the bootstrap CLI will generate default stack set administration role arns.
* The bootstrap CLI will deploy the stack with the parameter `TrustedStackSetAdminArns` set to a comma delimited list of the role arns.
This adds the stack set administration role arns to the assume role policy of the stack set execution role.
* Since the stack set execution role may have a large scope of permissions in the child account,
stack set administration role arns are used instead of account ids.
This scopes the assume role policy to specific roles versus any IAM entity in the parent account.

### CDK Bootstrap Template Changes

```yaml
Parameters:
# ...
EnableCreateStackSetRoles:
Description: Flag to create stack set roles
Default: false
Type: String
AllowedValues: [true, false]
TrustedStackSetAdminArns:
Description: List of stack set administration role arns that are trusted to
assume the stack set execution role
Default: ''
Type: CommaDelimitedList
# ...
Conditions:
# ...
CreateStackSetRoles:
Fn::Not:
- Fn::Equals:
- 'false'
- Ref: EnableCreateStackSetRoles
HasTrustedStackSetAdminArns:
Fn::Not:
- Fn::Equals:
- ''
- Fn::Join:
- ''
- Ref: TrustedStackSetAdminArns
# ...
Resources:
# ...
CloudFormationStackSetAdministrationRole:
Condition: CreateStackSetRoles
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Statement:
- Action: sts:AssumeRole
Effect: Allow
Principal:
Service: cloudformation.amazonaws.com
Version: '2012-10-17'
Policies:
- PolicyName:
Fn::Sub: cdk-${Qualifier}-stack-set-admin-role-policy-${AWS::AccountId}-${AWS::Region}
PolicyDocument:
Statement:
- Action:
- sts:AssumeRole
Resource:
- Fn::Sub: "arn:aws:iam::*:role/cdk-${Qualifier}-stack-set-exec-role-${AWS::Region}"
Effect: Allow
Version: '2012-10-17'
RoleName:
Fn::Sub: cdk-${Qualifier}-stack-set-admin-role-${AWS::AccountId}-${AWS::Region}
CloudFormationStackSetExecutionRole:
Condition: CreateStackSetRoles
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Statement:
- Action: sts:AssumeRole
Effect: Allow
Principal:
AWS:
Fn::GetAtt: CloudFormationStackSetAdministrationRole.Arn
- Fn::If:
- HasTrustedStackSetAdminArns
- Action: sts:AssumeRole
Effect: Allow
Principal:
AWS:
Ref: TrustedStackSetAdminArns
- Ref: AWS::NoValue
ManagedPolicyArns:
Fn::If:
- HasCloudFormationExecutionPolicies
- Ref: CloudFormationExecutionPolicies
- Fn::If:
- HasTrustedAccounts
# The CLI will prevent this case from occurring
- Ref: AWS::NoValue
# The CLI will advertise that we picked this implicitly
- - Fn::Sub: "arn:${AWS::Partition}:iam::aws:policy/AdministratorAccess"
RoleName:
# Role name does not contain account id since it must be identical across child accounts
Fn::Sub: cdk-${Qualifier}-stack-set-exec-role-${AWS::Region}
linsona marked this conversation as resolved.
Show resolved Hide resolved
# ...
```