Enhanced conflict review

[ccanash]

Support reviewing conflicts with Test Plan Runs for Test Plan Reports in the "Candidate" and "Recommended" phases

Problem

When conflicts are being evaluated it is done while the relevant AT + Browser combination for a Test Plan Version has not yet been marked as final (so still present on the Test Queue page). When there, it would allow testers and authors to go over the conflicts that testers have when recording results between their Test Runs.

Eventually, those conflicts are resolved between the testers’ runs and the reports are pushed forward to the CANDIDATE / RECOMMENDED phases. Eventually new reports may be created for the same combination of Test Plan Version + AT + Browser, but with different Browser / AT Version values and added to the Test Queue. But it would then mean the previous reports’ results are unable to be compared against the newly created report and that may be a desirable condition to have.

[boazsender]

@ccanash and @mcking65 How about we update the test-que with these changes to support this:

Add deep linking to each test in the test plan

Create a new test-conflicts ui at /test-que/:test-plan-report-id/conflicts

1. List of all tests in test-plan that have conflicts
2. Click on each test to open a disclosure that summarizes the conflicts and provides action buttons: a) Raise an issue, b) Admins: edit test-plan-report as a specific person (brings the admin to the existing test plan run page)
3. show issues raised based on conflicts in the relevant conflict section of this page

When a test-plan-report has conflict(s)

1. Show the count of conflicts per tests
2. Add a link to the test conflicts page

When a test-plan-report surfaces a conflict for an already recommend report, add it to the test queue, and show that it has conflicts

[css-meeting-bot]

The ARIA-AT Community Group just discussed Improving conflict resolution experience.

The full IRC log of that discussion

<jugglinmike> Topic: Improving conflict resolution experience
<jugglinmike> Boaz_Sender: We're coming up on being able to have the app re-run test plans in automation with new AT versions come out (or for other reasons)
<jugglinmike> Boaz_Sender: So there may be conflicts with automated test plan results
<jugglinmike> github: https://github.com//issues/975
<jugglinmike> Boaz_Sender: the changes being proposed here are a new "/conflicts" route on the app and also changes to the test queue to add an indication that there are conflicts
<jugglinmike> Matt_King: There's a heading here: "Create a new test-conflicts UI" and under it, there are two items
<jugglinmike> Matt_King: Is a path to this, then, from the test plan or from the report?
<jugglinmike> Matt_King: It reads, "Click on each test plan to open a disclosure [...]"
<jugglinmike> Boaz_Sender: In the test queue, there are test plans with test plan reports in them?
<jugglinmike> Matt_King: Right
<jugglinmike> Boaz_Sender: the proposal is to be able to click on the words that appear in the "status" column of the table (e.g. "one conflict")
<jugglinmike> Matt_King: So this would turn that into a link which goes where?
<jugglinmike> Boaz_Sender: To the new "/conflicts" page
<jugglinmike> Matt_King: And that would list all the tests in the test plan that have a conflict?
<jugglinmike> Boaz_Sender: Yes
<jugglinmike> Matt_King: Okay, so that's point #2
<jugglinmike> Matt_King: And the next heading reads "When a test-plan-report has conflict(s)"
<jugglinmike> Boaz_Sender: this would let you see who the conflicts are between
<jugglinmike> Matt_King: Right now, we have the situation where we have "report runs", but we don't surface each Tester's version of the data
<jugglinmike> Matt_King: We don't currently have a "report view" for each Tester's report
<jugglinmike> Matt_King: We present them in the test runner
<jugglinmike> Boaz_Sender: I don't think we necessarily need a "diff" view
<jugglinmike> Matt_King: We already have the summaries which are kind of like "micro-diffs"
<jugglinmike> Matt_King: I'm trying to think of the experience, though.
<jugglinmike> Matt_King: It says "add label to the test plan report", but right now, we don't have a report anywhere
<jugglinmike> Boaz_Sender: Let me change that to read, "Add label to the 'status' column of the test queue"
<jugglinmike> Matt_King: We already have a label there... Oh, but it doesn't say who the conflict is between
<jugglinmike> howard-e: I think we're trying to cover the situation where three Testers are assigned and there is a conflict between two of them
<jugglinmike> Matt_King: I don't know if this information is useful in the test queue
<jugglinmike> Matt_King: The conflicts could be between different groups of people
<jugglinmike> Boaz_Sender: Okay, I'll take that part out
<jugglinmike> Matt_King: When we're counting conflicts, can one test have multiple conflicts?
<jugglinmike> Matt_King: If one test can have more than one conflict, then we might want to say the number of tests in addition to the number of conflicts
<jugglinmike> howard-e: Yes, one test can have multiple conflicts. There could be conflicts between two different commands within a single test, for instance
<jugglinmike> Boaz_Sender: then there's "Add a link from the test-plan-report to the test-plan-report version history for that test plan in the data management section"
<jugglinmike> Matt_King: Right now, we have the "report status" dialog. Doesn't that always show up in data management?
<jugglinmike> howard-e: Oh, the link is already there. I think I misinterpreted the UI. We can remove this
<jugglinmike> Boaz_Sender: The last thing is, "When a test-plan-report surfaces a conflict for an already recommend report, add it to the test que, and show that it has conflicts"
<jugglinmike> IsaDC: I definitely could use a better way to get to the conflicts
<jugglinmike> IsaDC: My workflow involves searching for the word "conflict". It isn't great
<jugglinmike> Matt_King: I do the same thing!
<jugglinmike> Matt_King: One thing that would be awesome: if someone raises an issue, and it's tied to a conflict, it'd be great for that issue to be surfaced in this new "conflicts" page
<jugglinmike> Matt_King: The design of that "conflicts" page could probably use a wireframe.
<jugglinmike> Boaz_Sender: Would it be okay if we just implemented it and used that prototype as a basis for discussion?
<jugglinmike> Matt_King: Sure
<jugglinmike> Zakim, end the meeting