Bug: sam validate fails for template using 1.122.0 or 1.223.0

[dzwicker]

Description:

We have a template which validates and delays perfect using SAM CLI 1.121.0. But after upgrading it fails to validate with the error:

[[E0001: Error found when transforming the template] (Error transforming template: Key None must be a string) matched 1]
Error: Linting failed. At least one linting rule was matched to the provided template.

See template at the bottom!

I was able to 'debug' the template and the error seams to be a connector definition using type and are instead of an id.

eg:

    Connectors:
      AiWorkflowStabilisationFunctionS3Connector:
        Properties:
          Destination:
            Type: AWS::S3::Bucket
            Arn: !Sub arn:aws:s3:::qundo-legitimation-data-${Environment}
          Permissions:
          - Read
          - Write

Steps to reproduce:

sam validate --lint --template-file template.yaml --region eu-central-1

Observed result:

sam validate --template-file /Users/dzwicker/projects/QUNDO/ai/ai-workflows/workflows/photo/template.yaml --region eu-central-1 --debug --lint
2024-08-22 18:26:22,220 | No config file found in this directory.                                                                                                                                                                                                                                                  
2024-08-22 18:26:22,222 | OSError occurred while reading TOML file: [Errno 2] No such file or directory: '/Users/dzwicker/projects/QUNDO/ai/ai-workflows/workflows/photo/samconfig.toml'                                                                                                                           
2024-08-22 18:26:22,223 | Config file location: /Users/dzwicker/projects/QUNDO/ai/ai-workflows/workflows/photo/samconfig.toml                                                                                                                                                                                      
2024-08-22 18:26:22,223 | Config file '/Users/dzwicker/projects/QUNDO/ai/ai-workflows/workflows/photo/samconfig.toml' does not exist                                                                                                                                                                               
2024-08-22 18:26:22,238 | OSError occurred while reading TOML file: [Errno 2] No such file or directory: '/Users/dzwicker/projects/QUNDO/ai/ai-workflows/workflows/photo/samconfig.toml'                                                                                                                           
2024-08-22 18:26:22,238 | Using config file: samconfig.toml, config environment: default                                                                                                                                                                                                                           
2024-08-22 18:26:22,239 | Expand command line arguments to:                                                                                                                                                                                                                                                        
2024-08-22 18:26:22,239 | --template_file=/Users/dzwicker/projects/QUNDO/ai/ai-workflows/workflows/photo/template.yaml --lint                                                                                                                                                                                      
2024-08-22 18:26:22 Looking for CFLINTRC before attempting to load
2024-08-22 18:26:22 Validating User CFNLINTRC
2024-08-22 18:26:22 Validating CFNLINTRC config with given JSONSchema
2024-08-22 18:26:22 Schema used: {'$id': 'https://github.com/aws-cloudformation/cfn-python-lint/blob/main/src/cfnlint/data/CfnLintCli/config/schema.json', '$schema': 'http://json-schema.org/draft-07/schema#', 'additionalProperties': False, 'description': 'CFNLINTRC configuration schema', 'properties': {'append_rules': {'description': 'Location of directories to append rules from', 'items': {'type': 'string'}, 'type': 'array'}, 'configure_rules': {'additionalProperties': False, 'description': 'Configure rules', 'patternProperties': {'^.*$': {'patternProperties': {'^.*$': {'anyOf': [{'type': 'string'}, {'type': 'integer'}, {'type': 'boolean'}, {'items': {'type': 'string'}, 'type': 'array'}, {'items': {'type': 'integer'}, 'type': 'array'}, {'items': {'type': 'string'}, 'type': 'boolean'}]}}, 'type': 'object'}}, 'type': 'object'}, 'custom_rules': {'description': 'custom rule file to use', 'type': 'string'}, 'ignore_checks': {'description': 'List of checks to ignore', 'items': {'type': 'string'}, 'type': 'array'}, 'ignore_templates': {'description': 'Templates to ignore', 'items': {'type': 'string'}, 'type': 'array'}, 'include_checks': {'description': 'List of checks to include', 'items': {'type': 'string'}, 'type': 'array'}, 'mandatory_checks': {'description': 'List of mandatory checks to enforce', 'items': {'type': 'string'}, 'type': 'array'}, 'merge_configs': {'description': 'Merges lists between configuration layers', 'type': 'boolean'}, 'output_file': {'description': 'Path to the file to write the main output to', 'type': 'string'}, 'override_spec': {'description': 'Path to spec file to override with', 'type': 'string'}, 'regions': {'description': 'Regions to test against', 'items': {'type': 'string'}, 'type': 'array'}, 'registry_schemas': {'description': 'One or more directories of CloudFormation Registry Resource Schemas', 'items': {'type': 'string'}, 'type': 'array'}, 'templates': {'description': 'Templates to lint', 'items': {'type': 'string'}, 'type': 'array'}}, 'title': 'CFNLINTRC JSON Schema', 'type': 'object'}
2024-08-22 18:26:22 Config used: {}
2024-08-22 18:26:22 CFNLINTRC looks valid!
2024-08-22 18:26:22 Validating Project CFNLINTRC
2024-08-22 18:26:22 Validating CFNLINTRC config with given JSONSchema
2024-08-22 18:26:22 Schema used: {'$id': 'https://github.com/aws-cloudformation/cfn-python-lint/blob/main/src/cfnlint/data/CfnLintCli/config/schema.json', '$schema': 'http://json-schema.org/draft-07/schema#', 'additionalProperties': False, 'description': 'CFNLINTRC configuration schema', 'properties': {'append_rules': {'description': 'Location of directories to append rules from', 'items': {'type': 'string'}, 'type': 'array'}, 'configure_rules': {'additionalProperties': False, 'description': 'Configure rules', 'patternProperties': {'^.*$': {'patternProperties': {'^.*$': {'anyOf': [{'type': 'string'}, {'type': 'integer'}, {'type': 'boolean'}, {'items': {'type': 'string'}, 'type': 'array'}, {'items': {'type': 'integer'}, 'type': 'array'}, {'items': {'type': 'string'}, 'type': 'boolean'}]}}, 'type': 'object'}}, 'type': 'object'}, 'custom_rules': {'description': 'custom rule file to use', 'type': 'string'}, 'ignore_checks': {'description': 'List of checks to ignore', 'items': {'type': 'string'}, 'type': 'array'}, 'ignore_templates': {'description': 'Templates to ignore', 'items': {'type': 'string'}, 'type': 'array'}, 'include_checks': {'description': 'List of checks to include', 'items': {'type': 'string'}, 'type': 'array'}, 'mandatory_checks': {'description': 'List of mandatory checks to enforce', 'items': {'type': 'string'}, 'type': 'array'}, 'merge_configs': {'description': 'Merges lists between configuration layers', 'type': 'boolean'}, 'output_file': {'description': 'Path to the file to write the main output to', 'type': 'string'}, 'override_spec': {'description': 'Path to spec file to override with', 'type': 'string'}, 'regions': {'description': 'Regions to test against', 'items': {'type': 'string'}, 'type': 'array'}, 'registry_schemas': {'description': 'One or more directories of CloudFormation Registry Resource Schemas', 'items': {'type': 'string'}, 'type': 'array'}, 'templates': {'description': 'Templates to lint', 'items': {'type': 'string'}, 'type': 'array'}}, 'title': 'CFNLINTRC JSON Schema', 'type': 'object'}
2024-08-22 18:26:22 Config used: {}
2024-08-22 18:26:22 CFNLINTRC looks valid!
2024-08-22 18:26:22 User configuration loaded as
2024-08-22 18:26:22 {}
2024-08-22 18:26:22 Project configuration loaded as
2024-08-22 18:26:22 {}
2024-08-22 18:26:22 Merging configurations...
2024-08-22 18:26:22 Run scan of template None
2024-08-22 18:26:22 SAM Translator: 1.91.0
2024-08-22 18:26:22 Setting AWS_DEFAULT_REGION to eu-central-1
2024-08-22 18:26:22 Error transforming template: Key None must be a string
2024-08-22 18:26:22 Stack trace: Key None must be a string
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/aws-sam-cli/1.123.0/libexec/lib/python3.12/site-packages/cfnlint/template/transforms/_sam.py", line 156, in transform_template
    sam_translator.translate(
  File "/opt/homebrew/Cellar/aws-sam-cli/1.123.0/libexec/lib/python3.12/site-packages/samtranslator/translator/translator.py", line 188, in translate
    translated = macro.to_cloudformation(**kwargs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/aws-sam-cli/1.123.0/libexec/lib/python3.12/site-packages/samtranslator/metrics/method_decorator.py", line 117, in wrapper_cw_timer
    exec_result = func(*args, **kwargs)
                  ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/aws-sam-cli/1.123.0/libexec/lib/python3.12/site-packages/samtranslator/model/sam_resources.py", line 1909, in to_cloudformation
    self._add_connector_metadata(generated_resources, original_template, source, destination)
  File "/opt/homebrew/Cellar/aws-sam-cli/1.123.0/libexec/lib/python3.12/site-packages/samtranslator/model/sam_resources.py", line 2183, in _add_connector_metadata
    original_dest_type = original_resources.get(destination.logical_id, {}).get("Type")
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/aws-sam-cli/1.123.0/libexec/lib/python3.12/site-packages/cfnlint/decode/node.py", line 77, in get
    raise ValueError(f"Key {key!r} must be a string")
ValueError: Key None must be a string
[[E0001: Error found when transforming the template] (Error transforming template: Key None must be a string) matched 1]
2024-08-22 18:26:22,730 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics                                                                                                                                                                               
2024-08-22 18:26:22,731 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics                                                                                                                                                                               
2024-08-22 18:26:22,731 | Unable to find Click Context for getting session_id.                                                                                                                                                                                                                                     
Error: Linting failed. At least one linting rule was matched to the provided template.

Expected result:

Should be validated as before.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

template.yaml

AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31
Description: >
  ai-workflows-photo

Parameters:
  Environment:
    Type: String
    AllowedValues:
    - dev
    - staging
    - prod
    - dz

  LambdaRuntime:
    Type: String
    Default: provided.al2023
    AllowedValues:
    - provided.al2023
    - java21

  LambdaHandler:
    Type: String
    Default: not.used.in.provided.runtime
    AllowedValues:
    - not.used.in.provided.runtime
    - io.quarkus.amazon.lambda.runtime.QuarkusStreamHandler::handleRequest

  LambdaArchitecture:
    Type: String
    Default: arm64
    AllowedValues:
    - arm64
    - x86_64

  SnsTopicSubscription:
    Type: String
    Default: "[email protected]"

Globals:
  Function:
    Runtime: !Sub "${LambdaRuntime}"
    Architectures:
    - !Sub "${LambdaArchitecture}"
    Timeout: 20
    # 128 is ok but to be fast
    MemorySize: 128
    Handler: !Sub "${LambdaHandler}"

Resources:
  #Source SQS
  SourceQueue:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: !Sub ai-workflows-photo-queue-${Environment}

  #Result SQS
  ResultQueue:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: !Sub ai-workflows-photo-result-queue-${Environment}

  AiWorkflowStabilisationFunctionLog:
    Type: AWS::Logs::LogGroup
    Properties:
      RetentionInDays: 7
      LogGroupName: !Sub "/aws/lambda/ai-workflows-stabilisation-function-${Environment}"

  #Lambda Stabilisation
  AiWorkflowStabilisationFunction:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName: !Sub ai-workflows-stabilisation-function-${Environment}
      CodeUri: ../../lambdas/stabilisation/build/function.zip
      MemorySize: 1536
      Policies:
      - RekognitionLabelsPolicy: { }
    Metadata:
      SkipBuild: True
    # noinspection YamlFormatViolation
    Connectors:
      AiWorkflowStabilisationFunctionS3Connector:
        Properties:
          Destination:
            Type: AWS::S3::Bucket
            Arn: !Sub arn:aws:s3:::qundo-legitimation-data-${Environment}
          Permissions:
          - Read
          - Write
      AiWorkflowStabilisationFunctionS3ConnectorDev:
        Properties:
          Destination:
            Type: AWS::S3::Bucket
            Arn: arn:aws:s3:::qundo-legitimation-data-dev
          Permissions:
          - Read
          - Write
      AiWorkflowStabilisationFunctionS3ConnectorTest:
        Properties:
          Destination:
            Type: AWS::S3::Bucket
            Arn: arn:aws:s3:::qundo-kyc-test-data
          Permissions:
          - Read
          - Write

  AiWorkflowExtractionFunctionLog:
    Type: AWS::Logs::LogGroup
    Properties:
      RetentionInDays: 7
      LogGroupName: !Sub "/aws/lambda/ai-workflows-extraction-function-${Environment}"

  #Lambda Extraction
  AiWorkflowExtractionFunction:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName: !Sub ai-workflows-extraction-function-${Environment}
      CodeUri: ../../lambdas/extraction/build/function.zip
      MemorySize: 192
      Policies:
      - TextractDetectAnalyzePolicy: { }
      - TextractGetResultPolicy: { }
    Metadata:
      SkipBuild: True
    # noinspection YamlFormatViolation
    Connectors:
      AiWorkflowExtractionFunctionS3Connector:
        Properties:
          Destination:
            Type: AWS::S3::Bucket
            Arn: !Sub arn:aws:s3:::qundo-legitimation-data-${Environment}
          Permissions:
          - Read
      AiWorkflowExtractionFunctionS3ConnectorDev:
        Properties:
          Destination:
            Type: AWS::S3::Bucket
            Arn: arn:aws:s3:::qundo-legitimation-data-dev
          Permissions:
          - Read
      AiWorkflowExtractionFunctionS3ConnectorTest:
        Properties:
          Destination:
            Type: AWS::S3::Bucket
            Arn: arn:aws:s3:::qundo-kyc-test-data
          Permissions:
          - Read

  AiWorkflowExtractionMergeFunctionLog:
    Type: AWS::Logs::LogGroup
    Properties:
      RetentionInDays: 7
      LogGroupName: !Sub "/aws/lambda/ai-workflows-extraction-merge-function-${Environment}"

  #Lambda Merge
  AiWorkflowExtractionMergeFunction:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName: !Sub ai-workflows-extraction-merge-function-${Environment}
      CodeUri: ../../lambdas/extraction-merge/build/function.zip
    Metadata:
      SkipBuild: True

  #Logs for StepFunctions
  AiWorkflowStateMachineLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      RetentionInDays: 7
      LogGroupName: !Sub "/aws/snf/ai-workflows-${Environment}"

  AiWorkflowFaceComparisonFunctionLog:
    Type: AWS::Logs::LogGroup
    Properties:
      RetentionInDays: 7
      LogGroupName: !Sub "/aws/lambda/ai-workflows-face-comparison-function-${Environment}"

  #Lambda Face Comparision
  AiWorkflowFaceComparisonFunction:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName: !Sub ai-workflows-face-comparison-function-${Environment}
      CodeUri: ../../lambdas/face-comparison/build/function.zip
      Policies:
      - RekognitionFacesPolicy: { }
    Metadata:
      SkipBuild: True
    # noinspection YamlFormatViolation
    Connectors:
      AiWorkflowFaceComparisonFunctionS3Connector:
        Properties:
          Destination:
            Type: AWS::S3::Bucket
            Arn: !Sub arn:aws:s3:::qundo-legitimation-data-${Environment}
          Permissions:
          - Read
      AiWorkflowFaceComparisonFunctionS3ConnectorDev:
        Properties:
          Destination:
            Type: AWS::S3::Bucket
            Arn: arn:aws:s3:::qundo-legitimation-data-dev
          Permissions:
          - Read
      AiWorkflowFaceComparisonFunctionS3ConnectorTest:
        Properties:
          Destination:
            Type: AWS::S3::Bucket
            Arn: arn:aws:s3:::qundo-kyc-test-data
          Permissions:
          - Read

  AiWorkflowSnsTopic:
    Type: AWS::SNS::Topic
    Properties:
      TopicName: !Sub ai-workflows-error-${Environment}
      Subscription:
      - Endpoint: !Ref SnsTopicSubscription
        Protocol: "EMAIL"

  #Target StepFunction state machine
  AiWorkflowStateMachine:
    Type: AWS::Serverless::StateMachine
    Properties:
      Name: !Sub ai-workflow-photo-${Environment}
      DefinitionUri: workflow/ai-workflow-photo.asl.json
      DefinitionSubstitutions:
        AiWorkflowStabilisationFunctionArn: !GetAtt AiWorkflowStabilisationFunction.Arn
        AiWorkflowExtractionFunctionArn: !GetAtt AiWorkflowExtractionFunction.Arn
        AiWorkflowFaceComparisonFunctionArn: !GetAtt AiWorkflowFaceComparisonFunction.Arn
        AiWorkflowExtractionMergeFunctionArn: !GetAtt AiWorkflowExtractionMergeFunction.Arn
        AiWorkflowSnsTopicArn: !GetAtt AiWorkflowSnsTopic.TopicArn
      Logging:
        Destinations:
        - CloudWatchLogsLogGroup:
            LogGroupArn: !GetAtt AiWorkflowStateMachineLogGroup.Arn
        Level: ALL
        IncludeExecutionData: true
      Policies:
      - CloudWatchLogsFullAccess
      - Version: '2012-10-17' # Policy Document
        Statement:
        - Effect: Allow
          Action:
          - geo:SearchPlaceIndexForText
          Resource: 'arn:aws:geo:eu-central-1:001160930910:place-index/Qundo-Addresss-Check'
    # noinspection YamlFormatViolation
    Connectors:
      AiWorkflowStepFunctionStabilisationLambdaWriteConnector:
        Properties:
          Destination:
            Id: AiWorkflowStabilisationFunction
          Permissions:
          - Write
      AiWorkflowStepFunctionExtractionLambdaWriteConnector:
        Properties:
          Destination:
            Id: AiWorkflowExtractionFunction
          Permissions:
          - Write
      AiWorkflowStepFunctionFaceComparisonLambdaWriteConnector:
        Properties:
          Destination:
            Id: AiWorkflowFaceComparisonFunction
          Permissions:
          - Write
      AiWorkflowExtractionMergeLambdaWriteConnector:
        Properties:
          Destination:
            Id: AiWorkflowExtractionMergeFunction
          Permissions:
          - Write
      AiWorkflowStepFunctionResultQueueWriteConnector:
        Properties:
          Destination:
            Id: ResultQueue
          Permissions:
          - Write
      AiWorkflowStepFunctionResultQueueWriteConnectorDev:
        Properties:
          Destination:
            Type: AWS::SQS::Queue
            Arn: arn:aws:sqs:eu-central-1:001160930910:ai-workflows-photo-queue-dev
          Permissions:
          - Write
      AiWorkflowStepFunctionResultQueueWriteConnectorTest:
        Properties:
          Destination:
            Type: AWS::SQS::Queue
            Arn: arn:aws:sqs:eu-central-1:001160930910:ai-workflow-photo-result-for-test-*
          Permissions:
          - Write
      AiWorkflowStepFunctionSnsTopicWriteConnector:
        Properties:
          Destination:
            Id: AiWorkflowSnsTopic
          Permissions:
          - Write

  # Logs for EventBridge Pipe
  AiWorkflowSqsEventBridgePipeLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      RetentionInDays: 7
      LogGroupName: !Sub "/aws/pipe/ai-workflows-${Environment}"

  # Role for EventBridge Pipes to read from SQS and launch SFN
  AiWorkflowSqsEventBridgePipeRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Sub ai-workflow-sqs-event-bridge-pipe-role-${Environment}
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
        - Effect: Allow
          Principal:
            Service:
            - pipes.amazonaws.com
          Action:
          - sts:AssumeRole
      Policies:
      - PolicyName: CloudWatchLogs
        PolicyDocument:
          Version: 2012-10-17
          Statement:
          - Effect: Allow
            Action:
            - 'logs:CreateLogGroup'
            - 'logs:CreateLogStream'
            - 'logs:PutLogEvents'
            Resource: '*'
      - PolicyName: ReadSQS
        PolicyDocument:
          Version: 2012-10-17
          Statement:
          - Effect: Allow
            Action:
            - 'sqs:ReceiveMessage'
            - 'sqs:DeleteMessage'
            - 'sqs:GetQueueAttributes'
            Resource: !GetAtt SourceQueue.Arn
      - PolicyName: ExecuteSFN
        PolicyDocument:
          Version: 2012-10-17
          Statement:
          - Effect: Allow
            Action:
            - 'states:StartExecution'
            Resource: !Ref AiWorkflowStateMachine

  # EventBridge from SQS to StepFunction
  AiWorkflowSqsEventBridgePipe:
    Type: AWS::Pipes::Pipe
    Properties:
      Name: !Sub ai-workflow-photo-event-bridge-${Environment}
      RoleArn: !GetAtt AiWorkflowSqsEventBridgePipeRole.Arn
      DesiredState: RUNNING
      LogConfiguration:
        CloudwatchLogsLogDestination:
          LogGroupArn: !GetAtt AiWorkflowSqsEventBridgePipeLogGroup.Arn
        IncludeExecutionData:
        - ALL
        Level: TRACE
      Source: !GetAtt SourceQueue.Arn
      SourceParameters:
        SqsQueueParameters:
          BatchSize: 1
      Target: !Ref AiWorkflowStateMachine
      TargetParameters:
        StepFunctionStateMachineParameters:
          InvocationType: FIRE_AND_FORGET
        InputTemplate: |
          {
            "body": <$.body>
          }


Outputs:
  AiWorkflowSourceQueueUrl:
    Description: SourceQueue URL
    Value: !GetAtt SourceQueue.QueueUrl
  AiWorkflowResultQueueUrl:
    Description: ResultQueue URL
    Value: !GetAtt ResultQueue.QueueUrl
  AiWorkflowStateMachineLogGroup:
    Description: 'StepFunctions LogGroup Name'
    Value: !Ref AiWorkflowStateMachineLogGroup
  AiWorkflowStabilisationFunction:
    Description: "Stabilisation Lambda Function ARN"
    Value: !GetAtt AiWorkflowStabilisationFunction.Arn
  AiWorkflowExtractionFunction:
    Description: "Extraction Lambda Function ARN"
    Value: !GetAtt AiWorkflowExtractionFunction.Arn

[mildaniel]

In version 1.22.0 we bumped the cfn-lint version to 1.x #7207 which is likely the culprit.

It looks like the exception is coming from the SAM Transform being invoked by cfn-lint. It might be helpful to open a similar issue in the cfn-lint repository or the Transform Repository.

[mildaniel]

I did some more digging and it seems like I can reproduce the issue when invoking cfn-lint directly without SAM CLI too. I notice that the issue starts occurring when upgrading from cfn-lint version 0.87.10 to version 1.x.

Calling the SAM Transform directly I do not see the same error message which leads me to believe the issue stems from cfn-lint.

[hnnasit]

Closing this issue as I believe the PR to fix the above mentioned cfn-lint issue has been merged and released. The cfn-lint version bump was done as part of SAM CLI 1.124.0. Please open a new issue if you still have questions or concerns.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.