Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AWS::CertificateManager::Certificate - Region #523

Open
jk2l opened this issue Jun 15, 2020 · 35 comments
Open

AWS::CertificateManager::Certificate - Region #523

jk2l opened this issue Jun 15, 2020 · 35 comments
Labels

Comments

@jk2l
Copy link

jk2l commented Jun 15, 2020

1. Title

AWS::CertificateManager::Certificate - Region

Add new parameter for EDGE or REGIONAL (default REGIONAL).

Samples:

  SslCertificate:
    Type: AWS::CertificateManager::Certificate
    Properties:
      DomainName: domain.com
      SubjectAlternativeNames:
        - *.domain.com
      ValidationMethod: DNS
      Region: GLOBAL
      DomainValidationOptions:
        - DomainName: domain.com
          HostedZoneId: 123456789ABC

2. Scope of request

When I want to create one single stack with Cognito custom domain, or CloudFront I need to create the certificate in us-east-1 manually first. But I would like to have my certificate to be deployed from within the same CloudFormation template with my Cognito/ApiGateway...etc

Current behaviour for Cognito custom domain even if I deploy the Cfn stack in other regions such as ap-southeast-1, it is actually deployed it into us-east-1 as it is global resource via Edge location (if I am not mistaken).

However, this will require me to deploy the one environment in two separate regions. Having a centralized Cfn will allow management part so much easier.

The biggest issue of no support multi-region properly is that if there is a bug happen in the template, it cannot fully rollback properly, also rolling out a new update of certs (e.g. add a new alternative name) will change the ARN. if I have a multi-account environment (I have 10 environments planned). manage all ACM Arn one by one can add a lot of overhead and issue

3. Expected behaviour

With the additional parameter mark it as global, the certificate can be used by CloudFront. Even if everything deploys in a different region

4. Suggest specific test cases

as mentioned in section 3.

5. Helpful Links to speed up research and evaluation

  • Look for "Virginia" in the following docs

https://aws.amazon.com/premiumsupport/knowledge-center/custom-ssl-certificate-cloudfront/
https://docs.aws.amazon.com/cognito/latest/developerguide/cognito-user-pools-add-custom-domain.html
https://docs.aws.amazon.com/apigateway/latest/developerguide/how-to-custom-domains.html

Also for the sake of consistency, we can copy AWS::ApiGateway::RestApi EndpointConfiguration parameter format to use Edge, Regional as the option

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-apigateway-restapi-endpointconfiguration.html

6. Category (required) - Will help with tagging and be easier to find by other users to +1

  1. Compute
  2. Networking & Content
  3. Management
  4. Security
@jk2l
Copy link
Author

jk2l commented Jun 18, 2020

slightly updated the description to be inline and consistent with API gateway https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-apigateway-restapi-endpointconfiguration.html

instead of calling them GLOBAL and LOCAL. it should be called EDGE and REGIONAL

@rrrix
Copy link

rrrix commented Jun 18, 2020

Great idea!! 👍

We also just ran into this today while trying to setup a Cognito Custom Domain Name in us-west-2. I can't think of an elegant solution that doesn't require me managing 2 additional stacks or writing a custom resource Lambda to do the necessary cross-region stack deployment coordination for me.

@jk2l
Copy link
Author

jk2l commented Jul 17, 2020

Also suggested an alternative solution in #562

@max-allan-surevine
Copy link

See my answer to #630 . This is probably now possible with stacksets as cloudformation objects.

Your challenge is likely to be getting the properties of resources created in the sub-stacks.
I think your main stack would need to run in us-east-1 and create the ACM and then pass the ACM ID in to the nested stack, where the work happens in your other region. As long as you don't need to pass any information back from the region to us-east-1, I think you're OK. There is no "output" from the stackset.

@jk2l
Copy link
Author

jk2l commented Sep 24, 2020

hm... this can be more difficult to deploy into us-east-1. as I have big infrastructure stack that also passing around parameters from other stack's output. it will be much easier to deploy ACM as sub stack into us-east-1 instead of the other way round.

@jk2l
Copy link
Author

jk2l commented Sep 24, 2020

@max-allan-surevine how difficult will it be to introduce support for output references?

@max-allan-surevine
Copy link

max-allan-surevine commented Sep 25, 2020

I've just done it. It is pretty mind bending. I used some SSM params in the master region that a lambda in the sub-region can update. Then using the same concept and a custom resource type, that runs a lambda to update SSM.
If it is just one value you want to pass, shouldn't be too hard. If it is lots of values, I can imagine it getting very messy!! But for an ACM cert, it should be even easier than my example (that runs on 2 regions).
In this example, set reportValues.Input to whatever you want to "share" with the master (eg !Ref ACMCert)

example_region.txt

@aripalo
Copy link

aripalo commented Nov 2, 2020

Wanted to chime in and provide a use case for this feature request, since it would make things a lot easier for us.

Use case for Region: EDGE|REGIONAL

The challenge with multiple dynamic domains, AWS CloudFront, AWS Certificate Manager with infrastructure-as-code and other than us-east-1 as default region.

Background

DNS Setup

  • We develop multiple projects - think of APIs & services - that live under a "official basedomain", for example example.dev|.com.
  • We also utilize multiaccount strategy where each project/product/product-family (depends on the use case) has 2 AWS accounts:
    • dev
    • prod
  • Each account type has their own basedomain:
    • <project>.example.dev (dev-account)
    • <project>.example.com (prod-account)

domains


Environments (and CI/CD)

We use multiple environments (deployed via CI-service):

  • Main ones being:
    • production
    • staging
  • …but also dynamic preview environments, so a developer can actually deploy a new feature to AWS for testing and sharing it with stakeholders.
  • There's also - manually deployed - development environment which is mostly used for some random testing and during initial project setup.

cicd-environments

The challange

Our main region is eu-west-1 (Ireland), since it's in the EU, it's the close to us (Finland) and it's a feature rich region (compared to Stockholm).

We use AWS Cloud Development Kit (CDK) for infrastructure as code (CDK is based on CloudFormation).

The setup described above (in the diagrams) with environment specific domains and CI-controlled environments works like a charm otherwise, BUT it can be challenging to work with CloudFront and ACM:

  1. CloudFront requires that the certificate covers the alternate domain name on the same level as the CloudFront alias: So one can't use *.foo.example.dev in a CloudFront distribution with alias of api.my-cool-feature.preview.foo.example.dev.
  2. ... which means that each environment must create their own certificate, since we cannot know ahead of time about all the possibile subdomains.
  3. Creating DNS-validated ACM certificates in principle is easy with CloudFormation and CDK.
  4. ... but since we're deploying to eu-west-1 and CloudFront requires the certificate to exists in us-east-1, we need to use some kind of CloudFormation Custom Resource trickery.
  5. Luckily CDK provides acm.DnsValidatedCertificate for exactly that use case.
  6. ... unfortunately, since it may take 30 minutes for ACM to validate the certificate and the Lambda (that power Custom Resources) execution timeout is 15 minutes means that deployments often fail.
  7. Redeploying the stack again results in acm.DnsValidatedCertificate creating a new certificate (instead of using the certificate from previous deploy that probably is already validated), so the problem persists.
  8. One could use the acm.Certificate CDK construct, that will use native CloudFormation to create the certificate...
  9. ... BUT it's not an option, since CloudFormation cannot currently create a certificate to another region, remember that our main region is eu-west-1 and CloudFront requires ACM certificate in us-east-1!

So we're kind of stuck in this situation where new environment deployments randomly fail. We could probably create a separate CDK stack with acm.Certificate and deploy it to us-east-1 and then pass around the certificate ARN via Systems Manager parameter store, but that gets quite complex quite fast.

The perfect solution for us would be CloudFormation supporting certificate deployments to other regions, as proposed in CloudFormation roadmap issue #523 via Region-property (either REGIONAL or EDGE).

Also wondering if anyone else has hit this limitation with similar environment/DNS setup and how've they worked around the issue.

@Dzhuneyt
Copy link

Dzhuneyt commented Nov 2, 2020

@aripalo we have exactly the same use case minus the Git tag based production environments, and we often encounter the random stack deployments that you mentioned. Currently we found no feasible workaround besides creating a dedicated ACM stack in the us-east-1 region as you mentioned.

@andrewnicolalde
Copy link

It is very, very surprising that this isn't already supported given the requirement to use us-east-1 for API Gateway..

@aripalo
Copy link

aripalo commented Nov 24, 2020

Yep. The most depressing fact is that this seems not to be even on the roadmap (since no milestones etc is not assigned).

My guess is that most US-based users & folks at AWS don't see this as a problem since they're most probably deploying their workloads into us-east-1 anyway 🤷‍♂️

@jpluscplusm
Copy link

It is very, very surprising that this isn't already supported given the requirement to use us-east-1 for API Gateway..

I was gobsmacked, given that same requirement for CloudFront! I really hope this gets picked up sometime soon ...

@Purushotam-Thakur
Copy link

I'm also facing the same with my CloudFront, ACM as the other resources are in the eu-central-1 region while there is no option in serverless(CloudFormation) to change region while creating ACM.. any resolution or update on this??

@ankon
Copy link

ankon commented Feb 2, 2021

We hit this as well for a serverless API, and worked around the problem using a custom resource. This was a lot of work (and a lot more than expected), but seems to work reasonably well for now.

@jesusch
Copy link

jesusch commented Mar 5, 2021

why ain't ACM a global service since the very beginning...

@sirbully
Copy link

sirbully commented Mar 9, 2021

Experienced this problem too. Managed to come up with something based on @max-allan-surevine article. My issue now is how to get the output to show up from the stackset. I wanted to use the output to dynamically add it to my Cloudfront distribution but it's not showing up, neither is the export value that I tried to add. But it runs with no errors.

Certificate:
    Type: AWS::CloudFormation::StackSet
    Properties:
      Description: Request a certificate from ACM in N. Virginia
      PermissionModel: SELF_MANAGED
      StackInstancesGroup:
        - DeploymentTargets:
            Accounts:
              - !Ref AWS::AccountId
          Regions:
            - us-east-1
      StackSetName: !Sub ${AWS::StackName}-NorthVirginiaACM
      TemplateBody: |
        AWSTemplateFormatVersion: 2010-09-09
        Description: Request a certificate from ACM in N. Virginia
        Resources:
          USCertificate:
            Type: AWS::CertificateManager::Certificate
            Properties:
              DomainName: {redacted}
              SubjectAlternativeNames:
                - {redacted}
              DomainValidationOptions:
                - DomainName: {redacted}
                  HostedZoneId: {redacted}
              ValidationMethod: DNS
        Outputs:
          USCertificate:
            Value:
              Ref: USCertificate
            Export:
              Name: ami-Certificate

@lracicot
Copy link

I have the same issue. We have to deploy a stack that includes API Gateway with a custom domain using a CI/CD stack and we want to be able to freely deploy it in any region without manual intervention. We don't have capacity for developing/maintaining a custom solution. This is a huge pain point and the sole blocker for our current project.

@jk2l
Copy link
Author

jk2l commented Mar 18, 2021

why ain't ACM a global service since the very beginning...

lately I am more keen toward with my suggestion for #562 as I recently need to deploy a WAFv2 rule on top of the CloudFormation stack that is already deployed in ap-southeast-2. and found we come across the issue mentioned in #546

as this "region" deployment issue cover quite a bit different service. probably much more ideally to allow child stack to be deployed from same region and then refrence it's output

p.s. Solution from #630 is not a solution. as it does not allow refrencing output. so we will need to deploy custom solution on top of it which mean it back to square one (To have a solution that do not require hack like custom resource)

@mkamioner
Copy link

I think that solution #521 could answer this as well since it would essentially allow creation of certificates in multiple regions more easily.

@jk2l
Copy link
Author

jk2l commented May 27, 2021

I think that solution #521 could answer this as well since it would essentially allow creation of certificates in multiple regions more easily.

how is this related? what we want within this ticket is that the creation of ACM cert is in same region with resource that depends on it (e.g. if I create the CloudFront in with CloudFormation ap-southeast-2, I want to be able to create CloudFront's ACM from same stack as well).

but at the moment it is not possible to create ACM in us-east-1 from within ap-southeast-2 (but if you create global resource like CloudFront from ap-southeast-2, it is actually create us-east-1)

@WaelA WaelA added the Coverage label Aug 3, 2021
@danielmitchell
Copy link

Here's my solution, adapted from @max-allan-surevine.

My use case simply required an ACM certificate to be created in us-east-1 (I'm usually in a different region) for Cloudfront and the arn output in the main region so that another template can reference it. The stack set creates the certificate and uses a custom resource to pass the arn to a lambda function that outputs to an SSM parameter that can be read by the main template and output. The template also creates the required roles for stack sets to operate, which are needed even when running as an admin within the same account.

I ran into problems with the cfnresponse library not being available in the Lambda so refactored it to use the standard library with the identical return logic to correctly signal cfn. The custom resource passes two parameters Name and Value, where the name refers to the SSM parameter.

AWSTemplateFormatVersion: 2010-09-09

Parameters:
  DomainName:
    Default: test.something.com
    Type: String

Outputs:
  AppCertificateArnUsEastOutput:
    Value: !GetAtt AppCertificateArnUsEastParameter.Value
    Export:
      Name: AppCertificateArnUsEast

Resources:
  UsEastResources:
    Type: AWS::CloudFormation::StackSet
    Properties:
      StackSetName: !Sub "${AWS::StackName}-us-east-resources"
      PermissionModel: SELF_MANAGED
      StackInstancesGroup:
        - DeploymentTargets:
            Accounts:
              - !Ref AWS::AccountId
          Regions:
            - us-east-1
      TemplateBody: !Sub |
        AWSTemplateFormatVersion: 2010-09-09
        Resources:
          Certificate:
            Type: AWS::CertificateManager::Certificate
            Properties:
              DomainName: ${DomainName}
              ValidationMethod: DNS
          OutputFunction:
            Type: AWS::Lambda::Function
            Properties:
              FunctionName: stack-set-output
              Handler: index.lambda_handler
              Role: ${StackSetLambdaRole.Arn}
              Code:
                ZipFile: |
                  import boto3, json, urllib3
                  http = urllib3.PoolManager()

                  def lambda_handler(event, context):
                      print(f'Event: {json.dumps(event)}')
                      try:
                          client = boto3.client('ssm', region_name='${AWS::Region}')
                          client.put_parameter(Name=event['ResourceProperties']['Name'], Overwrite=True, Value=event['ResourceProperties']['Value'])
                          status = 'SUCCESS'
                      except Exception as e:
                          print(f'Failed to put parameter\n\nException: {e}')
                          status = 'FAILED'
                      json_response = json.dumps({'Status': status, 'PhysicalResourceId': context.log_stream_name, 'StackId': event['StackId'],
                                                  'RequestId': event['RequestId'], 'LogicalResourceId': event['LogicalResourceId']})
                      try:
                          response = http.request('PUT', event['ResponseURL'], body=json_response.encode('utf-8'),
                                                  headers={'content-type' : '', 'content-length' : str(len(json_response))})
                          print(f'Status code: {response.reason}')
                      except Exception as e:
                          print(f'Failed sending response\n\nResponse: {json_response}\n\nException: {e}')
              Runtime: python3.9
          OutputCertificateArn:
            Type: AWS::CloudFormation::CustomResource
            Properties:
              ServiceToken: !GetAtt OutputFunction.Arn
              Name: ${AppCertificateArnUsEastParameter}
              Value: !Ref Certificate
  
  AppCertificateArnUsEastParameter:
    Type: AWS::SSM::Parameter
    Properties:
      Name: /cloudformation/app-certificate-arn-us-east
      Type: String
      Value: none
  
  StackSetAdminRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: AWSCloudFormationStackSetAdministrationRole
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service: cloudformation.amazonaws.com
            Action:
              - sts:AssumeRole
      Policies:
        - PolicyName: AssumeRoleAWSCloudFormationStackSetExecutionRole
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - sts:AssumeRole
                Resource:
                  - arn:*:iam::*:role/AWSCloudFormationStackSetExecutionRole
  
  StackSetExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: AWSCloudFormationStackSetExecutionRole
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action:
              - sts:AssumeRole
            Principal:
              AWS:
                - !Ref AWS::AccountId
      ManagedPolicyArns:
        - !Sub arn:${AWS::Partition}:iam::aws:policy/AdministratorAccess
  
  StackSetLambdaRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: AWSCloudFormationStackSetLambdaRole
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action:
              - sts:AssumeRole
            Principal:
              Service:
                - lambda.amazonaws.com
      Policies:
        - PolicyName: AWSCloudFormationStackSetLambda
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - ssm:*
                Resource:
                  - !Sub arn:aws:ssm:${AWS::Region}:${AWS::AccountId}:parameter/cloudformation*
              - Effect: Allow
                Action:
                  - 'logs:*'
                # should be narrowed
                Resource: arn:aws:logs:*:*:*
            ```

@jk2l
Copy link
Author

jk2l commented Aug 26, 2021

tbh i am not a big fan of custom resources. debugging custom resources can sometimes be time-consuming. also, lambda version can get deprecate over the years so it requires some works to keep it up to date which this feature should be a native solution by default.

the major issue is the inconsistent design of cloud formation itself (e.g. you can create global resources like cloudfront from Sydney region, but you can't create an ACM associate it. )

Infrastructure as Code have been promoted from within AWS but the service itself failed to support a simple common use case.

@yosefbs
Copy link

yosefbs commented Feb 15, 2022

take a look at:
cross region cert

@carlin-q-scott
Copy link

@yosefbs Is this how it would be used in a template?

CloudFormationCert:
    type: AWS::CertificateManager::DnsValidatedCertificate
    Properties:
          DomainName: {redacted}
          HostedZone: {redacted}
          Region: us-east-1

@jk2l
Copy link
Author

jk2l commented Feb 17, 2022

@yosefbs Is this how it would be used in a template?

CloudFormationCert:
    type: AWS::CertificateManager::DnsValidatedCertificate
    Properties:
          DomainName: {redacted}
          HostedZone: {redacted}
          Region: us-east-1

lol no, CDK cheated it by using Lambda function custom resource. basically they wrap around some code logic to build a custom resource automatically on demand

@barankyle
Copy link

Here's my solution, adapted from @max-allan-surevine.

My use case simply required an ACM certificate to be created in us-east-1 (I'm usually in a different region) for Cloudfront and the arn output in the main region so that another template can reference it. The stack set creates the certificate and uses a custom resource to pass the arn to a lambda function that outputs to an SSM parameter that can be read by the main template and output. The template also creates the required roles for stack sets to operate, which are needed even when running as an admin within the same account.

I ran into problems with the cfnresponse library not being available in the Lambda so refactored it to use the standard library with the identical return logic to correctly signal cfn. The custom resource passes two parameters Name and Value, where the name refers to the SSM parameter.

AWSTemplateFormatVersion: 2010-09-09

Parameters:
  DomainName:
    Default: test.something.com
    Type: String

Outputs:
  AppCertificateArnUsEastOutput:
    Value: !GetAtt AppCertificateArnUsEastParameter.Value
    Export:
      Name: AppCertificateArnUsEast

Resources:
  UsEastResources:
    Type: AWS::CloudFormation::StackSet
    Properties:
      StackSetName: !Sub "${AWS::StackName}-us-east-resources"
      PermissionModel: SELF_MANAGED
      StackInstancesGroup:
        - DeploymentTargets:
            Accounts:
              - !Ref AWS::AccountId
          Regions:
            - us-east-1
      TemplateBody: !Sub |
        AWSTemplateFormatVersion: 2010-09-09
        Resources:
          Certificate:
            Type: AWS::CertificateManager::Certificate
            Properties:
              DomainName: ${DomainName}
              ValidationMethod: DNS
          OutputFunction:
            Type: AWS::Lambda::Function
            Properties:
              FunctionName: stack-set-output
              Handler: index.lambda_handler
              Role: ${StackSetLambdaRole.Arn}
              Code:
                ZipFile: |
                  import boto3, json, urllib3
                  http = urllib3.PoolManager()

                  def lambda_handler(event, context):
                      print(f'Event: {json.dumps(event)}')
                      try:
                          client = boto3.client('ssm', region_name='${AWS::Region}')
                          client.put_parameter(Name=event['ResourceProperties']['Name'], Overwrite=True, Value=event['ResourceProperties']['Value'])
                          status = 'SUCCESS'
                      except Exception as e:
                          print(f'Failed to put parameter\n\nException: {e}')
                          status = 'FAILED'
                      json_response = json.dumps({'Status': status, 'PhysicalResourceId': context.log_stream_name, 'StackId': event['StackId'],
                                                  'RequestId': event['RequestId'], 'LogicalResourceId': event['LogicalResourceId']})
                      try:
                          response = http.request('PUT', event['ResponseURL'], body=json_response.encode('utf-8'),
                                                  headers={'content-type' : '', 'content-length' : str(len(json_response))})
                          print(f'Status code: {response.reason}')
                      except Exception as e:
                          print(f'Failed sending response\n\nResponse: {json_response}\n\nException: {e}')
              Runtime: python3.9
          OutputCertificateArn:
            Type: AWS::CloudFormation::CustomResource
            Properties:
              ServiceToken: !GetAtt OutputFunction.Arn
              Name: ${AppCertificateArnUsEastParameter}
              Value: !Ref Certificate
  
  AppCertificateArnUsEastParameter:
    Type: AWS::SSM::Parameter
    Properties:
      Name: /cloudformation/app-certificate-arn-us-east
      Type: String
      Value: none
  
  StackSetAdminRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: AWSCloudFormationStackSetAdministrationRole
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service: cloudformation.amazonaws.com
            Action:
              - sts:AssumeRole
      Policies:
        - PolicyName: AssumeRoleAWSCloudFormationStackSetExecutionRole
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - sts:AssumeRole
                Resource:
                  - arn:*:iam::*:role/AWSCloudFormationStackSetExecutionRole
  
  StackSetExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: AWSCloudFormationStackSetExecutionRole
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action:
              - sts:AssumeRole
            Principal:
              AWS:
                - !Ref AWS::AccountId
      ManagedPolicyArns:
        - !Sub arn:${AWS::Partition}:iam::aws:policy/AdministratorAccess
  
  StackSetLambdaRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: AWSCloudFormationStackSetLambdaRole
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action:
              - sts:AssumeRole
            Principal:
              Service:
                - lambda.amazonaws.com
      Policies:
        - PolicyName: AWSCloudFormationStackSetLambda
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - ssm:*
                Resource:
                  - !Sub arn:aws:ssm:${AWS::Region}:${AWS::AccountId}:parameter/cloudformation*
              - Effect: Allow
                Action:
                  - 'logs:*'
                # should be narrowed
                Resource: arn:aws:logs:*:*:*
            ```

This was really helpful, though I needed to have UsEastResources DependsOn the StackSetAdminRole and StackSetExecutionRole so that it could be created and deleted with them around, otherwise there were errors. Also, I assume those triple ticks at the bottom were accidental extras for closing the markdown code section.

(I also added HostedZoneId as a parameter so that the cert could be automatically validated)

AWSTemplateFormatVersion: 2010-09-09

Parameters:
  DomainName:
    Type: String
  HostedZoneId:
    Type: String

Outputs:
  AppCertificateArnUsEastOutput:
    Value: !GetAtt AppCertificateArnUsEastParameter.Value
    Export:
      Name: AppCertificateArnUsEast

Resources:
  UsEastResources:
    Type: AWS::CloudFormation::StackSet
    DependsOn:
      - StackSetAdminRole
      - StackSetExecutionRole
    Properties:
      StackSetName: !Sub "${AWS::StackName}-us-east-resources"
      PermissionModel: SELF_MANAGED
      StackInstancesGroup:
        - DeploymentTargets:
            Accounts:
              - !Ref AWS::AccountId
          Regions:
            - us-east-1
      TemplateBody: !Sub |
        AWSTemplateFormatVersion: 2010-09-09
        Resources:
          Certificate:
            Type: AWS::CertificateManager::Certificate
            Properties:
              DomainName: ${DomainName}
              DomainValidationOptions:
                - DomainName: ${DomainName}
                  HostedZoneId: ${HostedZoneId}
              ValidationMethod: DNS
          OutputFunction:
            Type: AWS::Lambda::Function
            Properties:
              FunctionName: ${AWS::StackName}-stack-set-output
              Handler: index.lambda_handler
              Role: ${StackSetLambdaRole.Arn}
              Code:
                ZipFile: |
                  import boto3, json, urllib3
                  http = urllib3.PoolManager()

                  def lambda_handler(event, context):
                      print(f'Event: {json.dumps(event)}')
                      try:
                          client = boto3.client('ssm', region_name='${AWS::Region}')
                          client.put_parameter(Name=event['ResourceProperties']['Name'], Overwrite=True, Value=event['ResourceProperties']['Value'])
                          status = 'SUCCESS'
                      except Exception as e:
                          print(f'Failed to put parameter\n\nException: {e}')
                          status = 'FAILED'
                      json_response = json.dumps({'Status': status, 'PhysicalResourceId': context.log_stream_name, 'StackId': event['StackId'],
                                                  'RequestId': event['RequestId'], 'LogicalResourceId': event['LogicalResourceId']})
                      try:
                          response = http.request('PUT', event['ResponseURL'], body=json_response.encode('utf-8'),
                                                  headers={'content-type' : '', 'content-length' : str(len(json_response))})
                          print(f'Status code: {response.reason}')
                      except Exception as e:
                          print(f'Failed sending response\n\nResponse: {json_response}\n\nException: {e}')
              Runtime: python3.9
          OutputCertificateArn:
            Type: AWS::CloudFormation::CustomResource
            Properties:
              ServiceToken: !GetAtt OutputFunction.Arn
              Name: ${AppCertificateArnUsEastParameter}
              Value: !Ref Certificate
  
  AppCertificateArnUsEastParameter:
    Type: AWS::SSM::Parameter
    Properties:
      Name: /cloudformation/app-certificate-arn-us-east
      Type: String
      Value: none
  
  StackSetAdminRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: AWSCloudFormationStackSetAdministrationRole
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service: cloudformation.amazonaws.com
            Action:
              - sts:AssumeRole
      Policies:
        - PolicyName: AssumeRoleAWSCloudFormationStackSetExecutionRole
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - sts:AssumeRole
                Resource:
                  - arn:*:iam::*:role/AWSCloudFormationStackSetExecutionRole
  
  StackSetExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: AWSCloudFormationStackSetExecutionRole
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action:
              - sts:AssumeRole
            Principal:
              AWS:
                - !Ref AWS::AccountId
      ManagedPolicyArns:
        - !Sub arn:${AWS::Partition}:iam::aws:policy/AdministratorAccess
  
  StackSetLambdaRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: AWSCloudFormationStackSetLambdaRole
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action:
              - sts:AssumeRole
            Principal:
              Service:
                - lambda.amazonaws.com
      Policies:
        - PolicyName: AWSCloudFormationStackSetLambda
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - ssm:*
                Resource:
                  - !Sub arn:aws:ssm:${AWS::Region}:${AWS::AccountId}:parameter/cloudformation*
              - Effect: Allow
                Action:
                  - 'logs:*'
                # should be narrowed
                Resource: arn:aws:logs:*:*:*

@tfischer4765
Copy link

tfischer4765 commented Jun 9, 2022

Here's another use case to incentivise whomever it may concern to finally tackle this problem:

We have to deploy an application stack including a CloudFront Distribuition on a per-customer basis. That literally can't be deployed anywhere but the eu-central-1 region because it contains lambdas that need a link to a certain VPC containing a DB they need to access, which, for legal reasons, HAS to live in one of the eu-* regions.

Creating several hundred certs manually is right out of course.

We have however found an alternative workaround to the problem which may, especially for beginners, be easier to implement. We are using AWS API Gateway as a proxy to CloudFront.

This may not be an ideal solution especially performance-wise, and I would advise anyone who thinks about using it on a heavy-traffic site with global relevance to do their own research and tests, but it does the business for smaller, none too heavily frequented sites.

I'm providing a slightly snipped-down template in the hope that it might be useful to someone.

{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Transform": "AWS::Serverless-2016-10-31",
  "Description": "Sample SAM Template for using redirection through API Gateway to avoid the CloudFront certificate problem\n",
  "Globals": {
    "Function": {
      "Timeout": 30
    }
  },
  "Parameters": {
    "DomainName": {
      "Type": "String",
      "Default": "{redacted}"
    },
    "ApiHostedZone": {
      "Type": "AWS::Route53::HostedZone::Id",
      "Default": "{redacted}"
    },
    "BucketStackName": {
      "Type": "String",
      "Default": "{redacted}"
    }
  },
  "Resources": {
    "AccessLogs": {
      "Type": "AWS::Logs::LogGroup",
      "Properties": {
        "RetentionInDays": 7
      }
    },
    
    "HttpApi": {
      "Type": "AWS::Serverless::HttpApi",
      "Properties": {
        "AccessLogSettings": {
          "DestinationArn": {
            "Fn::GetAtt": [
              "AccessLogs",
              "Arn"
            ]
          },
          "Format": "$context.identity.sourceIp - - [$context.requestTime] \"$context.httpMethod $context.routeKey $context.protocol\" $context.status $context.responseLength $context.requestId $context.integration.integrationStatus $context.integration.error"
        },
        "FailOnWarnings": true,
        "CorsConfiguration": {
          "AllowOrigins": [
            {
              "Fn::Join": [
                "",
                [
                  "https://",
                  {
                    "Ref": "DomainName"
                  }
                ]
              ]
            }
          ],
          "AllowMethods": [
            "GET",
            "PUT",
            "OPTIONS"
          ],
          "MaxAge": 600
        },
        "Domain": {
          "CertificateArn": {
            "Ref": "FrontendCert"
          },
          "DomainName": {
            "Ref": "DomainName"
          },
          "Route53": {
            "HostedZoneId": {
              "Ref": "ApiHostedZone"
            }
          }
        },
        "DefinitionBody": {
          "openapi": "3.0.1",
          "info": {
            "title": {
              "Ref": "AWS::StackName"
            },
            "version": "1.0"
          },
          "paths": {
            "/$default": {
              "x-amazon-apigateway-any-method": {
                "isDefaultRoute": true,
                "x-amazon-apigateway-integration": {
                  "payloadFormatVersion": "1.0",
                  "type": "http_proxy",
                  "httpMethod": "ANY",
                  "uri": {
                    "Fn::Join": [
                      "",
                      [
                        "https://",
                        {
                          "Fn::GetAtt": [
                            "Distribution",
                            "DomainName"
                          ]
                        }
                      ]
                    ]
                  },
                  "connectionType": "INTERNET"
                }
              }
            }
          }
        }
      }
    },
    "FrontendCert": {
      "Type": "AWS::CertificateManager::Certificate",
      "Properties": {
        "DomainName": {
          "Ref": "DomainName"
        },
        "ValidationMethod": "DNS",
        "DomainValidationOptions": [
          {
            "DomainName": {
              "Ref": "DomainName"
            },
            "HostedZoneId": {
              "Ref": "ApiHostedZone"
            }
          }
        ]
      }
    },
    "Distribution": {
      "Type": "AWS::CloudFront::Distribution",
      "Properties": {
        "DistributionConfig": {
          "Origins": [
            {
              "DomainName": {
                "Fn::ImportValue": {
                  "Fn::Sub": "${BucketStackName}-StaticResourcesBucketUrl"
                }
              },
              "Id": "S3Origin",
              "S3OriginConfig": {
                "OriginAccessIdentity": {
                  "Fn::Join": [
                    "",
                    [
                      "origin-access-identity/cloudfront/",
                      {
                        "Fn::ImportValue": {
                          "Fn::Sub": "${BucketStackName}-OriginAccessIdentity"
                        }
                      }
                    ]
                  ]
                }
              }
            }
          ],
          "Enabled": true,
          "DefaultRootObject": "index.html",
          "DefaultCacheBehavior": {
            "AllowedMethods": [
              "DELETE",
              "GET",
              "HEAD",
              "OPTIONS",
              "PATCH",
              "POST",
              "PUT"
            ],
            "TargetOriginId": "S3Origin",
            "ForwardedValues": {
              "QueryString": false,
              "Cookies": {
                "Forward": "none"
              }
            },
            "ViewerProtocolPolicy": "redirect-to-https",
            "CachePolicyId": "658327ea-f89d-4fab-a63d-7e88639e58f6"
          },
          "PriceClass": "PriceClass_100",
          "ViewerCertificate": {
            "CloudFrontDefaultCertificate": true
          }
        }
      }
    }
  }
}

@peejayess
Copy link

peejayess commented Apr 13, 2023

Here's my solution, adapted from @max-allan-surevine.

My use case simply required an ACM certificate to be created in us-east-1 (I'm usually in a different region) for Cloudfront and the arn output in the main region so that another template can reference it. The stack set creates the certificate and uses a custom resource to pass the arn to a lambda function that outputs to an SSM parameter that can be read by the main template and output. The template also creates the required roles for stack sets to operate, which are needed even when running as an admin within the same account.

I ran into problems with the cfnresponse library not being available in the Lambda so refactored it to use the standard library with the identical return logic to correctly signal cfn. The custom resource passes two parameters Name and Value, where the name refers to the SSM parameter.

AWSTemplateFormatVersion: 2010-09-09

Parameters:
  DomainName:
    Default: test.something.com
    Type: String

Outputs:
  AppCertificateArnUsEastOutput:
    Value: !GetAtt AppCertificateArnUsEastParameter.Value
    Export:
      Name: AppCertificateArnUsEast

Resources:
  UsEastResources:
    Type: AWS::CloudFormation::StackSet
    Properties:
      StackSetName: !Sub "${AWS::StackName}-us-east-resources"
      PermissionModel: SELF_MANAGED
      StackInstancesGroup:
        - DeploymentTargets:
            Accounts:
              - !Ref AWS::AccountId
          Regions:
            - us-east-1
      TemplateBody: !Sub |
        AWSTemplateFormatVersion: 2010-09-09
        Resources:
          Certificate:
            Type: AWS::CertificateManager::Certificate
            Properties:
              DomainName: ${DomainName}
              ValidationMethod: DNS
          OutputFunction:
            Type: AWS::Lambda::Function
            Properties:
              FunctionName: stack-set-output
              Handler: index.lambda_handler
              Role: ${StackSetLambdaRole.Arn}
              Code:
                ZipFile: |
                  import boto3, json, urllib3
                  http = urllib3.PoolManager()

                  def lambda_handler(event, context):
                      print(f'Event: {json.dumps(event)}')
                      try:
                          client = boto3.client('ssm', region_name='${AWS::Region}')
                          client.put_parameter(Name=event['ResourceProperties']['Name'], Overwrite=True, Value=event['ResourceProperties']['Value'])
                          status = 'SUCCESS'
                      except Exception as e:
                          print(f'Failed to put parameter\n\nException: {e}')
                          status = 'FAILED'
                      json_response = json.dumps({'Status': status, 'PhysicalResourceId': context.log_stream_name, 'StackId': event['StackId'],
                                                  'RequestId': event['RequestId'], 'LogicalResourceId': event['LogicalResourceId']})
                      try:
                          response = http.request('PUT', event['ResponseURL'], body=json_response.encode('utf-8'),
                                                  headers={'content-type' : '', 'content-length' : str(len(json_response))})
                          print(f'Status code: {response.reason}')
                      except Exception as e:
                          print(f'Failed sending response\n\nResponse: {json_response}\n\nException: {e}')
              Runtime: python3.9
          OutputCertificateArn:
            Type: AWS::CloudFormation::CustomResource
            Properties:
              ServiceToken: !GetAtt OutputFunction.Arn
              Name: ${AppCertificateArnUsEastParameter}
              Value: !Ref Certificate
  
  AppCertificateArnUsEastParameter:
    Type: AWS::SSM::Parameter
    Properties:
      Name: /cloudformation/app-certificate-arn-us-east
      Type: String
      Value: none
  
  StackSetAdminRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: AWSCloudFormationStackSetAdministrationRole
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service: cloudformation.amazonaws.com
            Action:
              - sts:AssumeRole
      Policies:
        - PolicyName: AssumeRoleAWSCloudFormationStackSetExecutionRole
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - sts:AssumeRole
                Resource:
                  - arn:*:iam::*:role/AWSCloudFormationStackSetExecutionRole
  
  StackSetExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: AWSCloudFormationStackSetExecutionRole
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action:
              - sts:AssumeRole
            Principal:
              AWS:
                - !Ref AWS::AccountId
      ManagedPolicyArns:
        - !Sub arn:${AWS::Partition}:iam::aws:policy/AdministratorAccess
  
  StackSetLambdaRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: AWSCloudFormationStackSetLambdaRole
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action:
              - sts:AssumeRole
            Principal:
              Service:
                - lambda.amazonaws.com
      Policies:
        - PolicyName: AWSCloudFormationStackSetLambda
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - ssm:*
                Resource:
                  - !Sub arn:aws:ssm:${AWS::Region}:${AWS::AccountId}:parameter/cloudformation*
              - Effect: Allow
                Action:
                  - 'logs:*'
                # should be narrowed
                Resource: arn:aws:logs:*:*:*
            ```

Thanks very much for this. I did run into an issue using this to save the version of a AWS::Serverless::Function when AutoPublishAlias: live is set. Recommended resolutions can be found in aws/serverless-application-model#3079 (comment)

@rshayman
Copy link

+1. I can understand that it may not be wise to allow the creation of a certificate in any region, but surely you could add an attribute:

CertificateType: { Regional | Global }

that would allow from a global certificate for use with a Global resources such as CloudFront. Similarly, the ability to create a global LambdaFunction would simplify things greatly.

@jk2l
Copy link
Author

jk2l commented May 27, 2024

I have moved to aws CDK for last 2 years, it is ironically that there is a IaC required to put on top a IaC to solve something so simple. the AWS CDK introduce a custom resource (AWS Lambda) just to work around something to make basic feature supported

@laminarcode
Copy link

laminarcode commented Sep 14, 2024

A simpler request:
Since CloudFront, a non-regional construct, is available to be launched from a set of regions (outside us-east-1), then its pre-requisite components, such as TLS Certificates, must also be supported to be launched from these same set of regions.

Said differently: CloudFront seems to nicely handle itself when being launched outside us-east-1, it should also nicely handle a certificate, and perhaps Lambda @ Edge functions from the same regions.

Anything less is breaking CloudFormation requirements. Don't force the user to scratch the left ear with the right hand. This should be handled by the service.

AWS needs to move away from 'must be in us-east-1' concept as it breaks HA. Other services should follow the example of IAM when they need 'Global' services.

CloudFormation Custom resources are useful, but they are duct-tape not elegant solutions.

@cjgordon
Copy link

cjgordon commented Sep 23, 2024

Just spun the wheels on this one for a couple of hours until I found this gem: https://surevine.com/creating-cloudformation-stacks-in-multiple-aws-regions-with-common-resources/

Essentially add a substack to you stack just to create the cert in the right region:

    WebformCertificateSAM:
    Type: AWS::CloudFormation::StackSet
    Properties:
      StackSetName: webform-cert-sam
      PermissionModel: SELF_MANAGED
      StackInstancesGroup:
        - DeploymentTargets:
            Accounts:
              - !Ref "AWS::AccountId"
          Regions:
            - eu-west-1
      TemplateBody: !Sub
        - |
          AWSTemplateFormatVersion : '2010-09-09'
          Transform: AWS::Serverless-2016-10-31

          Parameters:
            StackId: 
              Type: String
            WebformUIDomain:
              Type: String

          Resources:

            WebformCertificate:
              Type: AWS::CertificateManager::Certificate
              Properties:
                DomainName: !Ref WebformUIDomain

          Outputs:
            WebformCertificateARN:
              Value: !Ref WebformCertificate
          

Then use the output in your other SAM/s. This seemed simpler than some of the other workarounds here.

@jk2l
Copy link
Author

jk2l commented Sep 23, 2024

Just spun the wheels on this one for a couple of hours until I found this gem: https://surevine.com/creating-cloudformation-stacks-in-multiple-aws-regions-with-common-resources/

Essentially add a substack to you stack just to create the cert in the right region:

    WebformCertificateSAM:
    Type: AWS::CloudFormation::StackSet
    Properties:
      StackSetName: webform-cert-sam
      PermissionModel: SELF_MANAGED
      StackInstancesGroup:
        - DeploymentTargets:
            Accounts:
              - !Ref "AWS::AccountId"
          Regions:
            - eu-west-1
      TemplateBody: !Sub
        - |
          AWSTemplateFormatVersion : '2010-09-09'
          Transform: AWS::Serverless-2016-10-31

          Parameters:
            StackId: 
              Type: String
            WebformUIDomain:
              Type: String

          Resources:

            WebformCertificate:
              Type: AWS::CertificateManager::Certificate
              Properties:
                DomainName: !Ref WebformUIDomain

          Outputs:
            WebformCertificateARN:
              Value: !Ref WebformCertificate
          

Then use the output in your other SAM/s. This seemed simpler than some of the other workarounds here.

creating cert isn't the problem. it's how to reference the ARN of the created resource. there is no native support to get the output value of AWS::CloudFormation::StackSet

For example, i want to create a Cognito domain in ap-southeast-2. the Cognito resource need to be create within ap-southeast-2 but the cert need to be create in us-east-1.

@jk2l
Copy link
Author

jk2l commented Jan 31, 2025

I have moved to aws CDK for last 2 years, it is ironically that there is a IaC required to put on top a IaC to solve something so simple. the AWS CDK introduce a custom resource (AWS Lambda) just to work around something to make basic feature supported

probably better to have this support officially as CDK seem to decide to deprecating the CDK custom resource for cross region module

@ironiridis
Copy link

ironiridis commented Mar 5, 2025

Here's a tested workaround using Secret region replication.

Note the line with the comment. You must reference the StackSet resource in the dynamic reference, otherwise CloudFormation will try to resolve the secret before it's been created and replicated.

AWSTemplateFormatVersion: "2010-09-09"

Parameters:
  DomainZoneId:
    Type: String
    Default: Z000000000EXAMPLE001
  Domain:
    Type: String
    Default: assets.yoursite.example

Resources:
  AssetCDN:
    Type: AWS::CloudFront::Distribution
    Properties:
      DistributionConfig:
        Enabled: true
        Aliases: [ !Sub "${Domain}" ]
        ViewerCertificate:
          AcmCertificateArn: !Sub
          - "{{resolve:secretsmanager:certificate-${Domain}-arn:SecretString}}"
          - Domain: !Ref Domain
            Stack: !Ref CDNCertificateStackSet # This is essential!
        Origins: [ /* ... */ ]
        DefaultCacheBehavior: { /* ... */ }

  CDNCertificateStackSet:
    Type: AWS::CloudFormation::StackSet
    Properties:
      CallAs: SELF
      PermissionModel: SELF_MANAGED
      StackInstancesGroup: [{ Regions: [ "us-east-1"], DeploymentTargets: { Accounts: [ !Ref AWS::AccountId ] } }]
      StackSetName: cdn-certificate-us-east-1
      TemplateBody: !Sub 
      - |
        AWSTemplateFormatVersion: "2010-09-09"
        Resources:
          Certificate:
            Type: AWS::CertificateManager::Certificate
            Properties:
              DomainName: "${Domain}"
              ValidationMethod: DNS
              DomainValidationOptions:
              - DomainName: "${Domain}"
                HostedZoneId: "${Zone}"
          CertificateArnSecret:
            Type: AWS::SecretsManager::Secret
            Properties:
              Description: "ARN of ACM Certificate for ${Domain}"
              Name: "certificate-${Domain}-arn"
              ReplicaRegions: [ { Region: "${Region}" } ]
              SecretString: !Ref Certificate
      - Domain: !Ref Domain
        Zone: !Ref DomainZoneId
        Region: !Ref AWS::Region

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests