[Feature Request] Add ability to Unit Test Templates

[shadycuz]

Now that #622 is merged it's a lot easier to work with Taskcat from inside python. During the process of developing the feature for functional testing from inside python I kept having problems with credentials. Setting them on the CLI and then refreshing them. It was a nightmare. It had me thinking about this whole testing thing and really the types of tests that I was writing were really only testing the Cloudformation service. Things like is X resource created or is "String" in Resource Name. None of these things need to be deployed to AWS be tested. What I was really doing was writing my unit tests as functional tests to make them work.

So I started thinking about what would it take to test a cloudformation template locally in order to do these unit tests.

1. Load all the template parameters either by taking user input or using the defaults provided for that parameter.
2. Resolve the conditionals
3. Resolve all intrinsic functions in the resources/outputs section.

The only somewhat tricky part here I found is when dealing with a nested intrinsic function you need to resolve from the deepest function and work your way out. It took 3 or 4 hours but I was able to build a rough POC. After a few days I had a nice working MVP. You can see how it works in the README under 'Usage'. The logic is quite simple and is all here. It's mostly ready to go except these few shortcomings:

• Implement all AWS intrinsic functions.
  • Only !Ref, !Sub, !Equals and !If currently supported.
• Add full functionality to pseudo variables.
  • Variables like Partition, URLSuffix should change if the region changes.
  • Variables like StackName and StackId should have a better default than ""
• Handle References to resources that shouldn't exist.
  • It's currently possible that a !Ref to a Resource stays in the final template even if that resource is later removed because of a conditional.

If the Taskcat maintainers would like this functionality I can begin porting it into taskcat.

@andrew-glenn

[andrew-glenn]

Hey Levi,

This sounds amazing. Would you mind pasting a couple quick blurbs?

I'm following on resolving the intrinsic functions, but I'm missing the bigger picture where they all tie in together. "Seeing" it has some serious mileage

[andrew-glenn]

Also - we have a Boto3 session manager that may help your credential nightmare in the future. You can get region objects - with sessions via ‘Config.get_regions()’

…

On Jan 27, 2021, at 07:51, Levi ***@***.***> wrote:  Now that #622 is merged it's a lot easier to work with Taskcat from inside python. During the process of developing the feature for functional testing from inside python I kept having problems with credentials. Setting them on the CLI and then refreshing them. It was a nightmare. It had me thinking about this whole testing thing and really the types of tests that I was writing were really only testing the Cloudformation service. Things like is X resource created or is "String" in Resource Name. None of these things need to be deployed to AWS be tested. What I was really doing was writing my unit tests as functional tests to make them work. So I started thinking about what would it take to test a cloudformation template locally in order to do these unit tests. Load all the template parameters either by taking user input or using the defaults provided for that parameter. Resolve the conditionals Resolve all intrinsic functions in the resources/outputs section. The only somewhat tricky part here I found is when dealing with a nested intrinsic function you need to resolve from the deepest function and work your way out. It took 3 or 4 hours but I was able to build a rough POC. After a few days I had a nice working MVP. You can see how it works in the README under 'Usage'. The logic is quite simple and is all here. It's mostly ready to go except these few shortcomings: Implement all AWS intrinsic functions. Only !Ref, !Sub, !Equals and !If currently supported. Add full functionality to pseudo variables. Variables like Partition, URLSuffix should change if the region changes. Variables like StackName and StackId should have a better default than "" Handle References to resources that shouldn't exist. It's currently possible that a !Ref to a Resource stays in the final template even if that resource is later removed because of a conditional. If the Taskcat maintainers would like this functionality I can begin porting it into taskcat. @andrew-glenn — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[shadycuz]

@andrew-glenn The credential issues are mainly around corp security. I have to do MFA to get a my tokens and then assume a proper role. That and worrying about my tokens expiring while running a long test meant I had to keep refreshing them etc. It's more of a general AWS / security thing then a taskcat problem.

I have examples in the read me but given he following template:

AWSTemplateFormatVersion: 2010-09-09
Description: "Creates an S3 bucket to store logs."

Parameters:
  BucketPrefix:
    Type: String
    Description: The name of the Application or Project using this bucket.
    MinLength: 2
    ConstraintDescription: "use only lower case letters or numbers"
    AllowedPattern: '[a-z0-9\-]+'
  KeepBucket:
    Type: String
    Description: Keep the bucket if the stack is deleted.
    AllowedValues:
      - 'TRUE'
      - 'FALSE'
    Default: 'FALSE'

Conditions:
  RetainBucket: !Equals [ !Ref KeepBucket, "TRUE" ]
  DeleteBucket: !Equals [ !Ref KeepBucket, "FALSE" ]

Resources:
  LogsBucket:
    Condition: DeleteBucket
    Type: AWS::S3::Bucket
    Properties:
      BucketName:
        !Sub ${BucketPrefix}-logs-${AWS::Region}

  RetainLogsBucket:
    Condition: RetainBucket
    Type: AWS::S3::Bucket
    DeletionPolicy: Retain
    Properties:
      BucketName:
        !Sub ${BucketPrefix}-logs-${AWS::Region}

Outputs:
  LogsBucketName:
    Description: Name of the logs bucket.
    Value: !If [RetainBucket, !Ref RetainLogsBucket, !Ref LogsBucket]
    Export:
      Name: !Sub ${BucketPrefix}-LogsBucket

This is the actual code I use to test:

from pathlib import Path

import pytest

from cloud_radar.unit_test import Template


@pytest.fixture
def template():
    template_path = Path(__file__).parent / "../templates/log_bucket/log_bucket.yaml"

    return Template.from_yaml(template_path.resolve())


def test_log_defaults(template):

    result = template.render({"BucketPrefix": "testing"})

    assert "LogsBucket" in result["Resources"]

    bucket_name = result["Resources"]["LogsBucket"]["Properties"]["BucketName"]

    assert "us-east-1" in bucket_name


def test_log_retain(template):

    result = template.render(
        {"BucketPrefix": "testing", "KeepBucket": "TRUE"}, region="us-west-2"
    )

    assert "LogsBucket" not in result["Resources"]

    bucket = result["Resources"]["RetainLogsBucket"]

    assert "DeletionPolicy" in bucket

    assert bucket["DeletionPolicy"] == "Retain"

    bucket_name = bucket["Properties"]["BucketName"]

    assert "us-west-2" in bucket_name

Now if I print the result variable from test_log_retain it prints:

AWSTemplateFormatVersion: 2010-09-09
Conditions:
  DeleteBucket: false
  RetainBucket: true
Description: Creates an S3 bucket to store logs.
Metadata:
  Cloud-Radar:
    Region: us-west-2
Outputs:
  LogsBucketName:
    Description: Name of the logs bucket.
    Export:
      Name: testing-LogsBucket
    Value: RetainLogsBucket
Parameters:
  BucketPrefix:
    AllowedPattern: '[a-z0-9\-]+'
    ConstraintDescription: use only lower case letters or numbers
    Description: The name of the Application or Project using this bucket.
    MinLength: 2
    Type: String
    Value: testing
  KeepBucket:
    AllowedValues:
    - 'TRUE'
    - 'FALSE'
    Default: 'FALSE'
    Description: Keep the bucket if the stack is deleted.
    Type: String
    Value: 'TRUE'
Resources:
  RetainLogsBucket:
    Condition: RetainBucket
    DeletionPolicy: Retain
    Properties:
      BucketName: testing-logs-us-west-2
    Type: AWS::S3::Bucket

The order is wrong but the values inside are correct ( I think 😉 )

EDIT: To sum it up, What I am doing is testing the logic inside the template itself instead of the results of the deployment. It's like having an offline version of the cloudformation service, though I understand you can't actually replace it 😝, this just gives you a chance to catch logical errors before you deploy. It also separates the unit testing from the functional testing quite nicely. It might seem scarry to create an offline cloudformation but most of the instrinctic functions are just native python functions so it turned out to no be much effort to get to where I am currently at. You can follow the path of development in the original PR DontShaveTheYak/cloud-radar#5

[andrew-glenn]

I think it makes total sense from the perspective of unit testing the templates. I'm not 100% certain how that translates back into the into the taskcat CLI approach of a deployment test.

However, I do think that it's a "build it and the use case will come" sort of thing... we may ask that the method within CFNTemplate change names - like last time - but otherwise, have at it

[shadycuz]

Its not really a CLI type thing 😉 , more of an import into python workflow. I can update the names and stuff. Should see a PR in the next week or two but it will take longer to complete than last time because of my day job.

[andrew-glenn]

Completely understand. I'm going to be bandwidth constrained as well, so the same caveat applies on this end as well.