(assert): Snapshot testing not working with 1.101.0 - directory references causes tests to diff #14468
Assignees
Labels
@aws-cdk/assert
Related to the @aws-cdk/assert package
bug
This issue is a bug.
needs-triage
This issue or PR still needs to be triaged.
response-requested
Waiting on additional info and feedback. Will move to "closing-soon" in 7 days.
Comments
markusl
added
bug
github-actions
bot
added
the
@aws-cdk/assert
|
Hi @markusl - I'm not sure I understand the test case vs the snapshot shown here. I believe the test case is trying to match the snapshot of the CloudFormation template, but the diff is not that of a CloudFormation template. Besides that, looking at the diff, the issue seems to be that the location of the |
nija-at
added
the
response-requested
|
@nija-at consider the following regression test case: test('TestStack', () => {
const app = new cdk.App({
context: {
'@aws-cdk/core:newStyleStackSynthesis': 'true',
},
outdir: 'cdk.out',
stackTraces: false,
});
// WHEN
const stack = new cdk.Stack(app, 'TestStack');
// THEN
expect(SynthUtils.toCloudFormation(stack)).toMatchSnapshot();
});With CDK up to 1.100.0 it generates the following snapshot and the regression test can be used in a CI system to check that nothing has changed: However, after updating to 1.101.0, the snapshot changes to a much larger one. This would not be a problem but it contains a part which changes every time when you run it in a CI system such as AWS CodeBuild: |
|
That diff makes a lot more sense now. It seems something changed that's impacting this conditional - aws-cdk/packages/@aws-cdk/assert-internal/lib/synth-utils.ts Lines 21 to 25 in 527f662 It was previously returning the template but now returns the entire synth output. My initial suspect would be this - b1ecd3d I'll need to dive into this more to confirm and identify the root cause. |
|
@markusl the reason is probably that there are two different instances of |
|
+1 to @rix0rrr. You've likely upgraded some of your CDK packages to 1.101.0 but not all. This will cause the |
|
@nija-at We did upgrade all of the main application CDK dependencies but we also have some third-party CDK constructs installed but we don't have control over them. Is there any other way to resolve this? I think it's unlikely that a project with three or more external dependencies would ever be perfectly synchronized regarding CDK library dependencies |
|
Can you confirm that this is the issue? Did you find multiple copies of |
|
@rix0rrr yes, that is correct: |
rix0rrr
added a commit
that referenced
this issue
May 5, 2021
The `assert` library sometimes fails to get the template out of a `CloudFormationStackArtifact` if there are multiple copies of the CDK in the dependency tree. Fix by replacing an `instanceof` with a duck-type check. Fixes #14468.
mergify bot
pushed a commit
that referenced
this issue
May 5, 2021
…14544) The `assert` library sometimes fails to get the template out of a `CloudFormationStackArtifact` if there are multiple copies of the CDK in the dependency tree. Fix by replacing an `instanceof` with a duck-type check. Fixes #14468. ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
|
john-tipper
pushed a commit
to john-tipper/aws-cdk
that referenced
this issue
May 10, 2021
…ws#14544) The `assert` library sometimes fails to get the template out of a `CloudFormationStackArtifact` if there are multiple copies of the CDK in the dependency tree. Fix by replacing an `instanceof` with a duck-type check. Fixes aws#14468. ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
|
I am afraid the same problem happens here: aws-cdk/packages/aws-cdk/lib/api/cloudformation-deployments.ts Lines 300 to 302 in b02311c |
hollanddd
pushed a commit
to hollanddd/aws-cdk
that referenced
this issue
Aug 26, 2021
…ws#14544) The `assert` library sometimes fails to get the template out of a `CloudFormationStackArtifact` if there are multiple copies of the CDK in the dependency tree. Fix by replacing an `instanceof` with a duck-type check. Fixes aws#14468. ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
|
Follow up on a similar issue (the one I mention just above): #17974 |
4 tasks
mergify bot
pushed a commit
that referenced
this issue
Mar 29, 2022
Fixes #17974 Follow-up of #14468 This follows the implementation of aws/constructs#955 to be more robust. ---- ### All Submissions: * [x] Have you followed the guidelines in our [Contributing guide?](https://github.com/aws/aws-cdk/blob/master/CONTRIBUTING.md) ### Adding new Unconventional Dependencies: * [ ] This PR adds new unconventional dependencies following the process described [here](https://github.com/aws/aws-cdk/blob/master/CONTRIBUTING.md/#adding-new-unconventional-dependencies) ### New Features * [ ] Have you added the new feature to an [integration test](https://github.com/aws/aws-cdk/blob/master/INTEGRATION_TESTS.md)? * [ ] Did you use `cdk-integ` to deploy the infrastructure and generate the snapshot (i.e. `cdk-integ` without `--dry-run`)? *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
StevePotter
pushed a commit
to StevePotter/aws-cdk
that referenced
this issue
Apr 27, 2022
Fixes aws#17974 Follow-up of aws#14468 This follows the implementation of aws/constructs#955 to be more robust. ---- ### All Submissions: * [x] Have you followed the guidelines in our [Contributing guide?](https://github.com/aws/aws-cdk/blob/master/CONTRIBUTING.md) ### Adding new Unconventional Dependencies: * [ ] This PR adds new unconventional dependencies following the process described [here](https://github.com/aws/aws-cdk/blob/master/CONTRIBUTING.md/#adding-new-unconventional-dependencies) ### New Features * [ ] Have you added the new feature to an [integration test](https://github.com/aws/aws-cdk/blob/master/INTEGRATION_TESTS.md)? * [ ] Did you use `cdk-integ` to deploy the infrastructure and generate the snapshot (i.e. `cdk-integ` without `--dry-run`)? *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
We utilize snapshots for a large amount of regression testing like this:
Reproduction Steps
See previous code.
What did you expect to happen?
Should so that regression testing using snapshots is possible.
What actually happened?
It seems that 1.101.0 has changed somehow and generates a new directory reference with every build which causes diffs:
- Snapshot - 2 + Received + 2 @@ -249,11 +249,11 @@ "libraries": Object {}, }, "version": "9.0.0", }, "bootstrapStackVersionSsmParameter": "/cdk-bootstrap/hnb659fds/version", - "file": "/Users/user/repo/cdk.out/Stack.assets.json", + "file": "/codebuild/output/src976733808/src/git-codecommit.eu-central-1.amazonaws.com/v1/repos/repo/cdk.out/Stack.assets.json", "id": "Stack.assets", "manifest": Object { "properties": Object { "bootstrapStackVersionSsmParameter": "/cdk-bootstrap/hnb659fds/version", "file": "Stack.assets.json", @@ -270,11 +270,11 @@ AssetManifestArtifact { "_dependencyIDs": Array [], "_deps": Array [], "assembly": [Circular], "bootstrapStackVersionSsmParameter": "/cdk-bootstrap/hnb659fds/version", - "file": "/Users/user/repo/cdk.out/Stack.assets.json", + "file": "/codebuild/output/src976733808/src/git-codecommit.eu-central-1.amazonaws.com/v1/repos/repo/cdk.out/Stack.assets.json", "id": "Stack.assets", "manifest": Object { "properties": Object { "bootstrapStackVersionSsmParameter": "/cdk-bootstrap/hnb659fds/version", "file": "Stack.assets.json", 63 | --Environment
Other
This is 🐛 Bug Report
The text was updated successfully, but these errors were encountered: