-
Notifications
You must be signed in to change notification settings - Fork 15
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Test Plans to the Test Queue: Plan version dropdown shows commits that do not include a change in the chosen test plan #579
Comments
Ready for Bocoup internal triage. It would be good to fix the dates that are shown in test queue. All should show December 8th |
This has been triaged and assigned to @alflennik |
Looking at the commit history for the alert test directory in the main branch of aria-at, I see the following versions for the alert plan:
In staging, When I open the add test plans to the test queue disclosure, choose alert, and look at the list of versions, the list is different. It is missing the Mar 17, 2022 version, and it has 3 others that are not in the above list: May 5, 2022, Mar 9, 2022, and Nov 18, 2021. I didn't yet make any other comparisons; I checked only alert. |
Using the sandbox on March 25, 2023.
When adding a plan to the test queue, the system is surfacing commits that do not include changes to the selected plan.
This enables generation of significant downstream data integrity problems.
Test plans are versioned by their last date of change as derived from the aria-at commit history.
The test plan version is critical information when running a plan through the working mode.
For example, it is esential to know what version of a plan was run to generate a specific result set and when the plan was last changed.
To reproduce the problem:
As of March 25, 2023, The last commit that change the
Alert Test Plan Files
was 0928bcf on December 8, 2022.
Actual behavior: Commits that do not include changes to the alert plan are listed in the version dropdown.
Expected behavior: Only commits to main with changes to the alert plan are listed.
Because of this problem, I was able to add multiple versions of the alert test plan to the test queue for VoiceOver with Chrome. The VoiceOver/Chrome table shows two rows for alert, one with a March 20 version and another with a March 8 version. This is a serious problem, because it indicates the plan has changed since December 8, but it has not changed. The ramifications of signaling change when in fact there was no change poses major risk to our ability to execute the working mode.
The text was updated successfully, but these errors were encountered: