sam validate will accept almost anything, never reports an error

[jelder]

Description:

I noticed that sam validate will print an error but then report success, including a zero exit code.

Steps to reproduce:

Create a template.yml full of errors, like this:

AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31

Parameters:
  WhateverParam:
    Type: whatever

Resources:
  WhateverResource:
    Type: whatever

Observed result:

❯ sam validate
2022-04-25 17:57:28 Loading policies from IAM...
2022-04-25 17:57:29 Finished loading policies from IAM.
2022-04-25 17:57:29 Template schema validation reported the following errors: [Parameters.WhateverParam.Type] 'whatever' is not one of ['String', 'Number', 'List<Number>', 'CommaDelimitedList', 'AWS::EC2::AvailabilityZone::Name', 'AWS::EC2::Image::Id', 'AWS::EC2::Instance::Id', 'AWS::EC2::KeyPair::KeyName', 'AWS::EC2::SecurityGroup::GroupName', 'AWS::EC2::SecurityGroup::Id', 'AWS::EC2::Subnet::Id', 'AWS::EC2::Volume::Id', 'AWS::EC2::VPC::Id', 'AWS::Route53::HostedZone::Id', 'List<AWS::EC2::AvailabilityZone::Name>', 'List<AWS::EC2::Image::Id>', 'List<AWS::EC2::Instance::Id>', 'List<AWS::EC2::SecurityGroup::GroupName>', 'List<AWS::EC2::SecurityGroup::Id>', 'List<AWS::EC2::Subnet::Id>', 'List<AWS::EC2::Volume::Id>', 'List<AWS::EC2::VPC::Id>', 'List<AWS::Route53::HostedZone::Id>', 'List<String>'], [Resources.WhateverResource.Type] 'whatever' must use format X::Y::Z
/Users/jacobelder/development/ironzion/template.yml is a valid SAM Template
❯ echo $?
0

Expected result:

I would expect that with validation errors, sam validate would:

1. Not print template.yml is a valid SAM Template
2. Exit with a nonzero status code.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: macOS 12.3.1
2. sam --version: SAM CLI, version 1.46.0
3. AWS region: us-east-1

❯ sam validate --debug
2022-04-25 17:58:10,911 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2022-04-25 17:58:10,911 | Using config file: samconfig.toml, config environment: default
2022-04-25 17:58:10,911 | Expand command line arguments to:
2022-04-25 17:58:10,911 | --template_file=/Users/jacobelder/development/ironzion/template.yml
2022-04-25 17:58:10 Loading policies from IAM...
2022-04-25 17:58:11 Finished loading policies from IAM.
2022-04-25 17:58:11 Template schema validation reported the following errors: [Parameters.WhateverParam.Type] 'whatever' is not one of ['String', 'Number', 'List<Number>', 'CommaDelimitedList', 'AWS::EC2::AvailabilityZone::Name', 'AWS::EC2::Image::Id', 'AWS::EC2::Instance::Id', 'AWS::EC2::KeyPair::KeyName', 'AWS::EC2::SecurityGroup::GroupName', 'AWS::EC2::SecurityGroup::Id', 'AWS::EC2::Subnet::Id', 'AWS::EC2::Volume::Id', 'AWS::EC2::VPC::Id', 'AWS::Route53::HostedZone::Id', 'List<AWS::EC2::AvailabilityZone::Name>', 'List<AWS::EC2::Image::Id>', 'List<AWS::EC2::Instance::Id>', 'List<AWS::EC2::SecurityGroup::GroupName>', 'List<AWS::EC2::SecurityGroup::Id>', 'List<AWS::EC2::Subnet::Id>', 'List<AWS::EC2::Volume::Id>', 'List<AWS::EC2::VPC::Id>', 'List<AWS::Route53::HostedZone::Id>', 'List<String>'], [Resources.WhateverResource.Type] 'whatever' must use format X::Y::Z
2022-04-25 17:58:11,731 | Translated template is:
AWSTemplateFormatVersion: '2010-09-09'
Parameters:
  WhateverParam:
    Type: whatever
Resources:
  WhateverResource:
    Type: whatever

/Users/jacobelder/development/ironzion/template.yml is a valid SAM Template
2022-04-25 17:58:11,732 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': '41ffaa57-6ac5-41e1-b43b-1413ff3aaa71', 'installationId': 'ae0f4e21-0be7-459f-8c45-5a73dbfd4374', 'sessionId': 'd23f0f8f-f49d-40f7-9302-705c389fa19a', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.8.13', 'samcliVersion': '1.46.0', 'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam validate', 'metricSpecificAttributes': {'projectType': 'CFN'}, 'duration': 820, 'exitReason': 'success', 'exitCode': 0}}]}
2022-04-25 17:58:12,084 | Telemetry response: 200

[jfuss]

@jelder The error printed is actually an printing error on. We attempted to add more validation within the SAM Spec and the output was not meant to be surfaced to CLI users yet as it's not complete. I thought we made changes to this to remove it but maybe that got missed. Will dig in further there.

sam validate can't actually validate the whole template in a meaningful way. So the output is actually expected if the SAM Spec is run without errors. Since the template doesn't have any SAM Resources, it will pass even if the template is not valid to CloudFormation. CloudFormation doesn't have a way to validate a full template either.

Will follow up on the error reporting but you can safely ignore that output.

[sriram-mv]

Valid concern, however with the release of cfn-lint and aws sam cli integration. One can run sam validate --lint and these errors should be picked up.

As of the latest release, sam init should create a default samconfig.toml file, this looks to be a great addition on that front.

sam validate --lint

W2001 Parameter WhateverParam not used.
/Users/srirammv/exp/perf-app/tt.yaml:5:3

E2002 Parameter WhateverParam has invalid type whatever
/Users/srirammv/exp/perf-app/tt.yaml:6:5

E3001 Invalid or unsupported Type whatever for resource WhateverResource in us-east-1
/Users/srirammv/exp/perf-app/tt.yaml:10:5

[mndeveci]

As @sriram-mv & @jfuss mentioned, SAM Spec has very basic validation which is mostly done to serverless resources. --lint option should give better validation of the CFN/SAM templates.

Resolving this issue but please let us know if you have further questions

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[dnemani]

sam validate does not point out basic syntax errors. As you can clearly see, this template is missing AWS: in the Principal:

            Principal:
                - !GetAtt LambdaFnServiceRole.Arn

But sam validate still does not catch it.

 sam validate --template-file pull_template.yaml --config-file samconfig-dev-us-east-1.toml --config-env pull --lint
pull_template.yaml is a valid SAM Template