-
Notifications
You must be signed in to change notification settings - Fork 403
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
RFC: Integration tests mechanism #1251
Comments
Should we really have a different provisioning model between E2E and integration tests? I think having to support two different models will he harder for maintainers and difficult to maintain in-sync. |
Suggestion: Creation/Tearing down of resources should be configurable for local tests. I can see how annoying it can be to have to run everything for a small changes that the developer knows don't have dependency on existing data (i.e. when running only specific tests in debugging a particular problem) |
Recommendation so we don't forget: The generated cloudformation should go through a cfn-lint/cfn-guard process before executing it. |
That's factored in. The only trade off we're making is to not have a different stack per utility -- this will slow down deployment/tear down considerably.
They're both CloudFormation. The differences are that E2E will use CDK to synthesise the various combinations of stacks, and use CFN to deploy; they create wildly different stacks, need a building process, packaging, permissions set and more. For Integ tests, we'll only create the resources our local codebase will integrate with -- a DynamoDB table, a SSM Parameter, a Secret -- no Lambda functions, no build or packaging process necessary. These will be considerably few resources making (up to 10x less than E2E). I personally find CDK an overkill for Integ tests in this context, but happy to hear it differently. The context switching is a good point tho - maybe it's worth exploring for one day to test ergonomics when we start implementation. |
Note to self for Monday before I forget: Experiment with stack per feature via conftest, it'll be easier to swap to CDK if we need to AND give us full test vs per feature step through debugging when we needed -- slower but more practical |
@heitorlessa This RFC looks pretty good! Few remarks from my side:
|
That’s stellar feedback - thank you! I’ll update the RFC this week and hold
any implementation until E2E settles the CDK approach.
One misconception though about verbose YAML. For Integ test, it’s easier to
maintain as the resources don’t have Config permutations and IAM
permissions. That said, context switching and cognitive load matters, so
I’ll revert to whatever E2E settles on.
…On Sun, 19 Jun 2022 at 20:44, mploski ***@***.***> wrote:
@heitorlessa <https://github.com/heitorlessa> This RFC looks pretty good!
Few remarks from my side:
1. In test classification It would be great to provide specific
example along with use case (as another column) - this will allow reader to
visualise it and clarify some typical confusion around testing: 1/ For
example functional testing may be understood as extension to integration
testing. 2/ Also e2e tests might be understood as rare tests that we want
to fire to cover most of the features/or most common feature from user
perspective, not specifically most complex one/the one with side effect.
2. In approach section I think we should also add points around
maintainability/readability. That was one of the point to use CDK SDK for
python to generate CF stack as mantainer/user can quickly read through it,
without need to dive deep in verbose json/yaml files.
3. Regarding point from @pcolazurdo <https://github.com/pcolazurdo>
about using one tool for integ/e2e tests I'm also in favour of this
approach:
1/ One way of spawning infrastructure regardless type of tests exposed
to mantainer/contributor will cause less confusion/ less maintenance effort.
2/ As long as we expose infra creation action as separate lib/fixture
with agreed interface for e2e testing, it should be straightforward to
reuse it directly in integration tests also
3/ Packaging and uploading overhead should be relatively low (although
we might measure it to confirm) as in integration tests we won't create any
additional assets like lambda or layer code) that needs to be deployed into
s3 bucket. The only thing CDK SDK will be responsible for is creating CF
stack yaml file in local environment
—
Reply to this email directly, view it on GitHub
<#1251 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAZPQBFQ6T5EW5EK22HYO3DVP5TBBANCNFSM5ZCWAYXQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Finally finished the E2E refactoring work. We can now use Integ tests with the same framework. Test definitions have been updated too - I'll update the RFC body as soon as I get a chance:
|
@mploski updated the RFC body and took the entire Looking at the docs, we'd highly likely only have Parameters, and maybe Feature Flags (uses Parameters Stores either way), since everything else is well covered in functional and E2E tests. |
I fixed that as part of the E2E refactoring work. Stacks are auto-deleted upon failures, you can use step through debugging without worrying about stack deletion, and you can also run the entire E2E suite in parallel locally w/o the need for CI. |
Thank you @heitorlessa. Whole doc looks great, no more remarks from my side :-) |
Hi everyone! We have the end2end tests running and covering various utilities that we have, which gives the project a sense of security that things are working as expected. Adding these integration tests in place would add more complexity to the project at this point. I am closing this issue. If there is a need in the future we can reopen it and continue the discussion/implementation. Thank you all for contributing with this RFC. |
|
Is this related to an existing feature request or issue?
#1009
Which AWS Lambda Powertools utility does this relate to?
Other
Summary
This RFC formalizes our thought process on integration tests (integ test). It also clarifies when functional, integration, and end-to-end tests are best suited.
Requirements
For security, integ tests will run on either scenarios: 1/ Locally targeting one's AWS Test Account, 2/ New and Existing PRs upon two approvers, and 3/ Existing PRs labelled
run-integ
after considered safe by maintainers.Use case
Integ tests can increase quality confidence in Powertools utilities with direct integration with AWS, e.g. Parameters, Feature Flags, etc.
Functional tests will remain useful to validate interfaces and pure Python logic. However, they can't help us detect regressions in contracts built on assumptions that integrations would remain backwards compatible.
NOTE: The majority of our tests will continue to be functional tests (subset of integ test) given speed and confidence ratio. Integ tests for extra confidence in direct integrations, and E2E for the complete picture we won't be able to simulate in either tests.
Proposal
Firstly, we formalize the definition of our tests. There will always be exceptions to the rule and we defer judgement to maintainers, as these can always be reverted if proven inefficient.
Test classification
Approach
We will follow the same approach as for E2E - Here's the most up to date structure.
Out of scope
This RFC doesn't cover E2E tests, nor contribution guide updates on which tests we will require.
Potential challenges
These are the immediate foreseen challenges:
Dependencies and Integrations
AWS CloudFormation to create resources required for each feature we want to integ test within a utility. Each utility will dictate which integration will be required.
Services we currently integrate regardless (excluding event sources):
Alternative solutions
We looked into AWS SDK for speed. However, given the number of integ test and the potential AWS cost it could incur customers in the event of stateful orphan resources (e.g., DynamoDB w/ several records), this became sub-optimal. We could create a static method to list/delete orphan resources for each resource we wanted to test, but this would need more thought on keeping state locally or created resources.
Terraform (TF) was a strong candidate given the speed and state management. Because E2E tests will be done in CloudFormation, it's more appropriate to stick with the same tool despite CDK usage in E2E tests. On a minor front, TF would need additional tools and docs for maintainers and contributors unfamiliar with it. We can easily reconsider this option, especially if we can achieve 50% greater speed.
Acknowledgment
The text was updated successfully, but these errors were encountered: