[Feature] Stack deployment file validation

[alnoki]

Is this feature request related to a new rule or cfn-lint capabilities?

Both

Describe the feature you'd like to request

Validation of stack deployment file contents:

• Enable deploy file discovery in pre-commit.
• Enable deploy/template file association.
• Verify all parameters specified in the associated template file are in the
  deployment file.
• Verify the deployment file does not contain parameters that are not in the
  specified template file.
• Verify the deploy file parameters comply with all template rules.

Describe the solution you'd like

For pre-commit, I currently have this block for cfn-lint:

-
  hooks:
  -
    # Lint all AWS CloudFormation files ending in `.cfn.yaml`.
    files: '.*\.cfn\.yaml'
    id: 'cfn-lint'
  repo: 'https://github.com/aws-cloudformation/cfn-lint'
  rev: 'v1.15.1'

Ideally there would be a way to flag deploy files, for example:

    # Lint all AWS CloudFormation deploy files ending in `.deploy.yaml`.
    deploy-files: '.*\.deploy\.yaml'

From there, cfn-lint could associate the template file from the deploy file,
which would look something like this:

---
parameters:
  DeployAnything: 'true'
  DeployGateway: 'true'
  DeployVpc: 'true'
tags: null
template-file-path: 'src/cloud-formation/template.cfn.yaml'
...

For this example, the folder src/cloud-formation/ has the above deploy file
at prod.deploy.yaml, which associates with template.cfn.yaml:

---
Conditions:
  DeployGateway: !Equals
  - !Ref 'DeployGateway'
  - 'true'
  DeployVpc: !Equals
  - !Ref 'DeployVpc'
  - 'true'
Parameters:
  DeployAnything:
    AllowedValues:
    - 'false'
    - 'true'
    Type: 'String'
  DeployGateway:
    AllowedValues:
    - 'false'
    - 'true'
    Type: 'String'
  DeployVpc:
    AllowedValues:
    - 'false'
    - 'true'
    Type: 'String'
Resources:
  InternetGateway:
    Condition: 'DeployGateway'
    Type: 'AWS::EC2::InternetGateway'
  InternetGatewayAttachment:
    Condition: 'DeployVpc'
    Properties:
      InternetGatewayId: !Ref 'InternetGateway'
      VpcId: !Ref 'Vpc'
    Type: 'AWS::EC2::VPCGatewayAttachment'
  Vpc:
    Condition: 'DeployVpc'
    Properties:
      CidrBlock: '0.0.0.0/16'
    Type: 'AWS::EC2::VPC'
Rules:
  DeployGateway:
    Assertions:
    - Assert: !Or
      - !Equals
        - !Ref 'DeployAnything'
        - 'true'
      - !Equals
        - !Ref 'DeployGateway'
        - 'false'
  DeployVpc:
    Assertions:
    - Assert: !Or
      - !Equals
        - !Ref 'DeployGateway'
        - 'true'
      - !Equals
        - !Ref 'DeployVpc'
        - 'false'
...

Given knowledge of the template, the deploy file parameters can then be
validated. The above example deploy file would pass, however the following
cases should fail:

Missing a parameter:

---
parameters:
  DeployGateway: 'true'
  DeployVpc: 'true'
tags: null
template-file-path: 'src/cloud-formation/template.cfn.yaml'
...

Contains erroneous parameter:

---
parameters:
  DeployAnything: 'true'
  DeployGateway: 'true'
  DeployVpc: 'true'
  Foo: 'bar'
tags: null
template-file-path: 'src/cloud-formation/template.cfn.yaml'
...

Parameters violate rules:

---
parameters:
  DeployAnything: 'false'
  DeployGateway: 'true'
  DeployVpc: 'true'
tags: null
template-file-path: 'src/cloud-formation/template.cfn.yaml'
...

Additional context

This is closely tied to fixes made by @kddejong to close recent issues:

• [Bug] Impossible, rules-unaware condition errors #3630
• [Bug] Rules support Condition, which is not a rule-specific intrinsic function #3710

Is this something that you'd be interested in working on?

I can provide input and extra supporting content, but I'm not familiar with the cfn-lint codebase.

Would this feature include a breaking change?

This feature should not incur a breaking change

[kddejong]

I like the idea of being able to use parameter files. The git-sync format is probably the best format as it associates the parameters and the template file to be able to do this work. I think second would be able to support the AWS CLI parameter key/value file.

It would really cut down on alternative if/then/else processing we do. I would love to efficiently do this so we aren't doing duplicate processing of things that aren't changing.

[whoDoneItAgain]

I think it would also be beneficial to support the configuration json file used by codepipeline. It would involve some kind of mapping file as the codepipeline stage declarations tells cloud formation which template and parameter file to use for each stack. In my environment I'll use the same cf template with many jsons in the same pipeline.

[kddejong]

Allow cfn-lint to use deployment files to lint and fill in parameters

• Original Author(s):: @kddejong

Summary

Allow cfn-lint to use deployment files instead of directly providing templates. Deployment files will contain paths to template and parameter files. cfn-lint will use these files to validate different templates and parameter scenarios.

Example

Given the template:

Parameters:
  Environment:
    Type: String
    AllowedValues:
    - production
    - stage
    - development
  BucketName:
    Type: String
Resources:
  Bucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Ref BucketName
      Tags:
      - Key: Environment
        Value: !Ref Environment

and a deployment file production.yaml

template-file-path: template.yaml
parameters:
  Environment: production
  BucketName: foo*bar

and a deployment file development.yaml

template-file-path: template.yaml
parameters:
  Environment: dev
  WrongParameter: my-bucket

We would get errors like:

EXXXX 'foo*bar' does not match '^[a-z0-9][a-z0-9.-]*[a-z0-9]$' when resolved by deployment file 'production.yaml'
template.yaml:14:7
EXXXX 'dev' is not one of ['production', 'stage', 'development']
development.yaml:3:2
EXXXX 'WrongParameter' is not one of ['Environment', 'BucketName']
development.yaml:4:2
EXXXX 'BucketName' is a required property
development.yaml:2:1

Motivation

Currently, cfn-lint validates Default and AllowedValues for parameters, which can result in validating more scenarios than intended. Using deployment files would:

• Limit the number of validation scenarios
• Validate parameter value correctness
• Verify parameter existence in templates

Details

• Add a new cfn-lint configuration parameter called --deployment-files which can take a glob compatible string to any number of deployment files. This parameter is exclusive with templates as this may cause confusion with the runner.
• Expand the runner capabilities to include deployment files.
• Create a pluggable system to handle deployment files and create the initial plugin for Git sync deployment files.
• Process the template replacing Ref to parameters with their appropriate value
• Process any functions that can be fully resolved. If they can't be fully resolved we leave them as they are.
• Run rules against the resolved template
  • Note if we process the templates replacing parameters for Ref values certain rules will have to be disabled. Example I3042, W3010, W8003, etc.

Limitation

• Resolution of Pseudo-Parameters may be limited depending on the information in the deployment file.
  • AWS::AccountId, AWS::Region, AWS::Partition, AWS::StackId, AWS::StackName, AWS::NotificationArns, AWS::URLSuffix

Outstanding questions

• While cfn-lint has made great strides for how it handles parameter combinations this may not be efficient. Ideally we would only change how a template works as we run into a parameter value however this can be complicated. It may be easier to process a template with the parameter values then lint the processed template.
• Have to create a mechanism to validate the correctness of deployment files. One method is to create a json schema document based on the template then use that schema to validate the deployment file. From the example this will cover any of the errors that we display in the deployment files themselves.

References

• CloudFormation Git sync
• CodePipeline CloudFormation action
• GitHub CloudFormation deploy