chore: Runs tests for latest minor versions of each major TF version #1769
Conversation
lantoli
force-pushed
the
CLOUDP-218618_test_suite
branch
from
74e1df0 to
71c50d4
Compare
| inputs: | ||
| terraform_matrix: | ||
| description: 'Terraform matrix version (JSON array)' | ||
| default: '["1.6.x", "1.5.x", "1.4.x", "1.3.x", "1.2.x"]' |
There was a problem hiding this comment.
I checked that this notation works, e.g. 1.4.x will run version 1.4.7.
TF version list is here: https://releases.hashicorp.com/terraform/
| @@ -1 +1,2 @@ | |||
| golang 1.21.3 | |||
| terraform 1.6.6 | |||
There was a problem hiding this comment.
opportunistic change not related to the main goal of the PR
lantoli
changed the title
|
|
||
| concurrency: | ||
| group: '${{ github.workflow }}' | ||
| cancel-in-progress: false |
There was a problem hiding this comment.
only one Test Suite run at a time to avoid interference. new runs will wait until previous ones finish
| versions: | ||
| env: | ||
| schedule_terraform_matrix: '["1.6.x", "1.0.8"]' | ||
| schedule_provider_matrix: '["", "1.12.3]' # "" for latest version |
There was a problem hiding this comment.
schedule triggers can't have input parameters so we define the values we want here
| default: '["1.6.x", "1.5.x", "1.4.x", "1.3.x", "1.2.x"]' | ||
| provider_matrix: | ||
| description: 'MongoDB Atlas Provider version matrix (JSON array)' | ||
| default: '["", "1.12.3"]' # "" for latest version |
There was a problem hiding this comment.
should we still have 1.12.3 hardcoded here? I think what we want ultimately is that our latest version is compatible... right @Zuhairahmed ?
There was a problem hiding this comment.
@marcosuma Zuhair told me to check our latest 2 minor patch versions, "" is the latest one (currently 1.13.1).
for example once we release 1.14.0, we'll have to change this to: "", "1.13.1", where "" will be 1.14.0
There was a problem hiding this comment.
I don't understand why 😅 and also this is something we'll have to manually update too often, every time we make a release (yet one more thing to do manually). Unless there is a strong reason (let's wait for Zuhair) that I am missing let's please remove it.
There was a problem hiding this comment.
Feedback is to test last 2 Majors of Terraform Atlas Provider versions (BigMajor.Major.Minor format) so today the latest minor patch on each would be v1.13.1 and v1.12.3. This is to get ahead of any unintended breaking changes before our customers were to identify them so we can do this by ensuring the same tests pass last 2 Major versions. Would however explore ways to prevent hardcoding here if possible.
There was a problem hiding this comment.
is it just me or I am really not seeing the value in this?
Once you've gone over 1.12.3, what is the point in keep testing the second last version? That will never change because we will never touch it.
To me it's more the extra manual effort we are introducing in having to edit the last version every time than the benefit of this
There was a problem hiding this comment.
would consider the HashiCorp Terraform testing scenarios where:
- acceptance tests fail on Terraform Atlas Provider v1.13.1 and pass on v1.12.3
- acceptance tests fail on both Terraform Atlas Provider v1.13.1 and v1.12.3
if we only tested most recent version all we would know for certain is that tests have failed on v1.13.1 and we need to investigate why this occurred which can be from multiple sources. however by running testing on most 2 recent recent version of Terraform Atlas Provider we help accelerate triage of finding a bug by narrowing scope of issue. if scenario 1 one occurs, then we know that error is something that we introduced with most recent changes catching breaking changes ahead of customers. if it's scenario 2, then we know that issue most likely has to do with change made on HashiCorp Terraform side (for example they released bug) so we can then escalate to HashiCorp thereby saving time with bug resolution.
Does this help? Agree again that we should be automating/avoiding hardcodes as much of this as possible.
There was a problem hiding this comment.
removing previous comment
Ok, I got some updates from Leo offline, so I think the main misunderstanding here is that these are migration tests, hence 1.12.3 is used to first create the resource and then "latest" is used to check if there are no plan changes.
Let us discuss this offline.
| provider_matrix: | ||
| description: 'MongoDB Atlas Provider version matrix (JSON array)' | ||
| default: '["", "1.12.3"]' # "" for latest version | ||
|
|
||
| schedule: | ||
| - cron: "0 0 * * *" # workflow runs every day at midnight UTC |
There was a problem hiding this comment.
can we do longer frequency? every month?
There was a problem hiding this comment.
this frequency is for:
- schedule_terraform_matrix: '["1.6.x", "1.0.8"]'
- schedule_provider_matrix: '["", "1.12.3]' # "" for latest version
the longer version test has to be run manually, for instance we can do it monthly or before a release or a major release.
There was a problem hiding this comment.
If we look at the requirement we discussed yesterday, we want to test our latest version against the terraform versions that we declare we support. This was also stated in one of the document Zuhair has recently proposed.
So my ask is: can we please have a monthly-based testing that checks our latest provider version against '["1.6.x", "1.5.x", "1.4.x", "1.3.x", "1.2.x"]'. I'd prefer not to do this manually as we have already lots of things we do manually.
Let me know if I am missing something
There was a problem hiding this comment.
we could but the problem I see is that somebody has to look at the results, and if it's done once per month I see it very easy to miss that execution result. Also I find it interesting to run it at other occasions like before a release.
this is also happening with the daily Test Suite. We run them automatically every night, but you have to go to the GH UI to see the results.
If we run something monthly I think that execution will probably be missed.
I don't see the benefit of running it monthly automatically. I think it's better something like I did today: I run them manually and then I look at the results.
There was a problem hiding this comment.
that said, I you still think it's a good idea please confirm and I'll add the monthly schedule
There was a problem hiding this comment.
On this, I don't have a strong opinion, I am ok to run it manually or before a release. @gssbzn how do we do it for AtlasCLI?
There was a problem hiding this comment.
how do we do it for AtlasCLI?
We always run packaging on master (we call it the waterfall on evergreen), this means after every merge, and then all the OS compatibility depends on packaging succeeding. The compatibility matrix exploded on the CLI at some point and evergreen let's us generate test programmatically so we write the yaml via code here
This reverts commit 5f82991.
Co-authored-by: Agustin Bettati <[email protected]>
|
Description
Runs tests for latest minor versions of each major TF version.
Link to any related issue(s): https://jira.mongodb.org/browse/CLOUDP-218618
Type of change:
Required Checklist:
Further comments