SSM Dynamic Reference Change not detected

[robot-apocalypse]

When using SSM Dynamic references, if the reference or resolved value changes, cloudformation does not detect the change and will not update the stack.

Scenario 1: changing {{resolve:ssm:/dev/pipeline/GitHubOwner}} to {{resolve:ssm:/production/pipeline/GitHubOwner}} in the template

Scenario 2: changing {{resolve:ssm:/dev/pipeline/GitHubOwner:10}} to {{resolve:ssm:/dev/pipeline/GitHubOwner:11}} in the template

Scenario 3: changing the value of the parameter without changing the template (using new 'latest' functionality)

I would expect all three scenarios to be recognized as a change and allow the stack to be updated, instead I get
"No changes to deploy. Stack production-pipeline is up to date"

[sulmowski]

I can confirm that issue.

I created lambda with resolve ssm as a key object in cloudformation template. Then I deployed the stack.

SomeLambda:
    Type: 'AWS::Lambda::Function'
    Properties:
      Code:
        S3Bucket: !Sub '${lambdaBucketName}-${stage}'
        S3Key: !Sub '{{resolve:ssm:/${stage}/SomeLambda}}'

Any updates later couldformation does not see any changes perhaps that parameter store has a new value and version.

[kddejong]

@robot-apocalypse are you doing an update stack while using the existing template? I was getting different results based on if I supplied a template or used the existing one. I'm curious if you also tried both approaches or not.

[robot-apocalypse]

@kddejong I genuinely don't remember, I opened this ticket because the original issue (#75) was fixed really badly (IMHO). This has been a problem since 2019. I had switch to terraform a while ago

[kftsehk]

Had a discussion with support and heard this wasn't fixed yet.

Resources:
  MyResource
    Properties:
      SomeProperty: !Sub "{{resolve:ssm:${SomeParameterOrValue}}}"

Had a workaround, plug a random source (e.g. timestamp) into metadata to force a change set

Parameters:
  RandomSource:
    Type: String
Resources:
  MyResource
    Metadata:
      WorkaroundSubSSM: !Ref "RandomSource"
    Properties:
      SomeProperty: !Sub "{{resolve:ssm:${SomeParameterOrValue}}}"

Of course would very like to hear that this get fixed, and don't have to litter the Metadata everywhere in the template.

[ALutchko]

How can we push this bug up?

[IyadKandalaft]

+1

[msclbu]

+1

[kftsehk]

I thought it should be a simple fix in change set creation, like when the syntax is "{{resolve:ssm:...}}" / !Sub "{{resolve:ssm:...}}" / anything evaluated to "{{resolve:ssm:...}}", assume there will be a change, which is exactly what the above metadata workaround does.

It is incorrect to assume that if the text "{{resolve:ssm:....}}" does not change, no change set has to be created. The value resolved from SSM can change, which seem to be checked only after a change set is created.

It haven't been fixed for another half year, and our team had written a simple github action that checks if every resource that used !Sub {{resolve:ssm}} has Metadata.WorkaroundSubSSM to ensure nothing got missed.

jobs:
  schema:
    name: WorkaroundSubSSM
    runs-on: ubuntu-latest
    steps:
      - name: Set up Python
        uses: actions/setup-python@v2
        with:
          architecture: 'x64'
          python-version: '3.x'
      - name: Install dependencies
        run: python -m pip install yq
      - name: Install jq
        run: sudo apt-get install jq
      - name: Check out Git repository
        uses: actions/checkout@v3
        with:
          fetch-depth: 0
      - name: Check metadata field
        run: |
          MISSING_METADATA=0
          for f in $(git ls-files '*.yml' '*.yaml')
          do
            isCfnTemplate=$(yq '.AWSTemplateFormatVersion' "$f")
            [ "$isCfnTemplate" == "null" ] && continue
            # find Resource entry with {{resolve::ssm}} but not Metadata.WorkaroundSubSSM
            # shellcheck disable=SC2016
            resourceTags=$(yq -r '
              .Resources | to_entries[] | if (
                .value | [
                  leaf_paths as $path | {
                    "key": $path | join("_"),
                    "value": getpath($path)
                  }
                ] | from_entries | select (
                  [.[] | strings ] | .[] | test(".resolve.")
                ) | keys | map (
                  select(
                    . | contains("Metadata_WorkaroundSubSSM")
                  )
                ) | any
              ) then empty else .key end' "$f")
            if [ -n "$resourceTags" ]; then
              for tag in $resourceTags;
              do
                line=$(grep -n "^ \+$tag:$" "$f" | cut -d : -f 1)
                MISSING_METADATA=1
                echo "::error file=$f,line=$line,title=Missing WorkaroundSubSSM::$f:L$line: Resource missing Metadata.WorkaroundSubSSM when using {{resolve:ssm}}"
              done
            fi
          done
          if [ "$MISSING_METADATA" -eq 1 ]
          then
            exit 1
          fi
name: CloudFormation
'on':
  pull_request:
    branches:
      - main
      - dev
  push:
    branches:
      - main
      - dev

[ALutchko]

I understood the idea with Metadata, I don't like it because it causes rebuilding every time and when you have a lot of resources it takes really a lot of time. My workaround: I use just regular parameters instead of "{{resolve:ssm:....}}"and then gather all the values where the template is being run (e.g. in CodeBuild). This way we do make sure values are actual and at the same time do not trigger rebuild when it is not needed.

[msclbu]

I believe I have a workaround for Secrets Manager (haven't tested with SSM).

As the VersionId is updated every time you perform an update to a secret, you can use the AWS SDK/CLI to grab that VersionID and append it your dynamic reference in your CF template. This is stated in the document as version-id https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/dynamic-references.html#dynamic-references-secretsmanager-pattern

For example I could run this CLI query:

(aws secretsmanager list-secret-version-ids
--secret-id YourSecret --query
'Versions[?contains(VersionStages, AWSCURRENT) == true].VersionId' --output text)

That would grab the latest VersionID that matches the AWSCURRENT version stage and them pass that through as a parameter to your template, looking something like this:

Resource: - !Sub '{{resolve:secretsmanager:arn:aws:secretsmanager:ap-southeast-2:${Account}:secret:${YourSecret}:SecretString:YourSecretString:AWSCURRENT:${SecretVersion}}}}'

[benbridts]

There is one more thing you need to know, on top of the solution with the random/updated value.

If you have this code, the parameter will still not update a second time

Parameters:
  RandomSource:
    Type: String
Resources:
  MyResource
    Metadata:
      WorkaroundSubSSM: !Ref "RandomSource"
    Properties:
      SomeProperty: !Sub "{{resolve:ssm:/hardcoded/parameter/path}}"

In most case you will not run into this, as either you have a dynamic value you're using in the !Sub, or you are using a Type: AWS::SSM::Parameter::Value<String> Parameter

For reference, if you want to hardcoded a value with a resolve (should only be needed during testing), you can write.

Parameters:
  RandomSource:
    Type: String
Resources:
  MyResource
    Metadata:
      WorkaroundSubSSM: !Ref "RandomSource"
    Properties:
      SomeProperty: !Sub
        - "{{resolve:ssm:${Parameter}}}"
        - Parameter: /hardcoded/parameter/path

[Arushi1597]

The introduction of this GitHub issue seems a little misleading to me. Scenario 1 and Scenario 2 mentioned in this Github issue seem to detect the changes just fine. I am not sure why its mentioned that these two scenarios do not detect change. Following is my observation for these three scenarios:

---

Scenario 1: changing {{resolve:ssm:test}} to {{resolve:ssm:new}} in the template using:

!Sub '{{resolve:ssm:${parametername}}}'

Outcome: ChangeSet created successfully/ Update successful

---

Scenario 2: changing {{resolve:ssm:test:1}} to {{resolve:ssm:test:2}} in the template using:

!Sub '{{resolve:ssm:${parametername}:${version}}}’

Outcome: ChangeSet created successfully/ Update successful

---

Scenario 3: Using ssm dynamic reference without parameter version like below in order to fetch latest value and then changing the ssm parameter value in SSM without changing anything in the template:

!Sub '{{resolve:ssm:${parametername}}}'

Outcome: ChangeSet creation fails/ Update fails with error: No updates are to be performed.

---

Its only in Scenario 3 i.e. when the value of the SSM parameter is changed without changing the template that the Changeset creation fails. This happens because CFN does not have a way at that time to check if the resolved value of the dynamic reference has changed or not as dynamic references are resolved when you execute the change set (not at the time of change set creation), as mentioned in the AWS official doc.

So in order to trigger a stack update in this case and to also fetch the latest parameter version from parameter store during that stack update, it becomes important to update the resource containing the dynamic reference, either by updating the resource property that contains the dynamic reference, or by updating another of the resource's properties.

Now, one way to do that is by using parameter versions in the dynamic reference in the template and have a system in place to match the latest version of the parameter in the parameter store with the version present in the template. If this method is not suitable, then the following workaround can also come handy for this use-case:

• Use SSM parameter types in place of SSM dynamic references. An update to the below stack (without any changes to the template) also retrieves the latest value from the parameter store.

Parameters:
  parametername:
    Type: AWS::SSM::Parameter::Value<String>
  
Resources:  
  MyS3Bucket:
    Type: 'AWS::S3::Bucket'
    Properties:
      BucketName: !Ref parametername

[kftsehk]

There is one more thing you need to know, on top of the solution with the random/updated value.

If you have this code, the parameter will still not update a second time

Parameters:
  RandomSource:
    Type: String
Resources:
  MyResource
    Metadata:
      WorkaroundSubSSM: !Ref "RandomSource"
    Properties:
      SomeProperty: !Sub "{{resolve:ssm:/hardcoded/parameter/path}}"

@benbridts Hi, you can then remove the unnecessary !Sub and it works fine

@Arushi1597 Yes, it is only case 3 that didn't work, am testing again now and still not fixed.

[sergiy2303]

I expected case 3 to work, but it doesn't. Is it so hard to resolve ssm parameters and after that create change set with new latest version?

[a-tom]

I can only second what @Arushi1597 wrote in Scenario 3.
I see the problem that between the change set creation and execution the parameter might have changed as well, but it's better to have a change set, than being told that nothing has changed.

[nqdrizzt]

I have the same issue.

Dynamic ssm references are not getting resolved during a change set creation and therefore end up in "no changes found" although the value of the latest version of a ssm parameter hast changed.

eg:
For an EC2 instance the following property is set via SSM parameterstore:
IamInstanceProfile: !Sub "{{resolve:ssm:/${projectID}/iam/Ec2InstanceProfileName}}"

But when I change the value of the parameter Ec2InstanceProfileName in the SSM Parameter store and create a change set for the CF stack, which manages the EC2 instance, there is no change set created (and therfor the EC2 instance not updated), becasue CF does not determine a change in the template.

[kftsehk]

It is likely a problem with the way changeset is created, hypothesis here is based on the fact that if you skip changeset creation and do an update, the actual parameter gets updated, and also adding random metadata will cause a correct update.

ChangeSet = NewResolvedTemplate - OldResolvedTemplate (Likely stateless)

Dynamic reference is not a part of template resolution, so remains as {{resolve:ssm:...}}, now

NewResolvedTemplate -> {{resolve:ssm:...<no-version-id>}} -> resolves to current value
OldResolvedTemplate  -> {{resolve:ssm:...<no-version-id>}} -> it did not record the previous value, so recomputing the resolve also retrieves the same current value

So no change detected.

As long as you make some change, even to the metadata, an actual cloudformation update to the resource is performed, there seem no such diff issue above in the update implementation

• If there is no change in SSM parameter, updating metadata in cloudformation is a small overhead
• If there is change in SSM parameter, the resource is updated