Improve PR review experience

[andrross]

The most important information on the front page of a PR is the author's description of the change and the reviewers' comments. However, we have made liberal use of the "create-or-update" GitHub action for our various workflows to post information directly on the PR. This is all well intentioned and there is important information being posted, but I believe we've probably gone to far and the single-to-noise ratio here has dropped, and too often comments are obscured by the dreaded "load more" button. Also my scroll wheel finger is getting quite the workout :) Here are some specific thoughts I have to improve things:

• Move to a "no news is good news" model. If an action succeeds, I think the green checkmark at the bottom of the page is sufficient. No need to make a comment indicating success.
• Dedupe and cancel workflows when PR updates are pushed. Workflows can be slow, and it is quite common to push a commit, find a problem, then push another commit a few minutes later. Currently both of these commits will complete a gradle check run and both runs will report a result as a comment.
• Change the "check" and "check compatibility" workflows to update a single comment with the latest results instead of only appending (not sure if this is a good idea to be honest)

Any other ideas?

[BhumikaSaini-Amazon]

+1 for this, especially true for critical / long-running PRs.

I haven't tested on the feasibility of this yet, but does it make sense to create a new GitHub issue (with relevant tags added) for each PR (similar to the auto backport PRs for example), wherein all of the GitHub actions can post the results etc.? The issue description could house the latest info from the most recent action runs. On the PR side, there could be a validation to resolve open comments on the linked issue before merging the PR. This would also keep the PR conversation relatively more clean with only the review comments displayed.

99% of the times, we are only looking for the results from the most recent runs. Its only when we want to maybe fetch a specific test seed or go back for another issue we observed in a previous build/action that we actually go looking for the previous build/action results.

[dblock]

I like the simple proposal of removing green noise!

Another idea is for gradle check to be deleting, or editing the previous gradle check comment, regardless of its status, so you only see the latest. This is something that danger does.

[andrross]

does it make sense to create a new GitHub issue (with relevant tags added) for each PR

@BhumikaSaini-Amazon This is an interesting idea, but I'm inclined not to do this because the information that is posted in comments is already available elsewhere (in the list of github action workflows at the bottom of the PR), it just takes several clicks to get to the relevant information. The reason we post comments is to highlight the information and make it accessible with fewer clicks, I just think we've gone way too far with this.

[andrross]

gradle check to be deleting, or editing the previous gradle check comment

@dblock I think this is possible with the create-or-update workflow we're using (see example). The biggest challenge might be race conditions because it is rather important that only the latest run update the single comment.

[andrross]

The summary here is that the "check compatibility" workflow has been updated to update a single comment, bringing it in line with the codecov behavior, which should substantially reduce noise on the PR comment feed. However, multiple contributors mentioned that they prefer that "gradle check" create a new comment for every run so that it generates a notification that the PR needs attention (either success or failure). There were also opinions that it is desirable to not cancel concurrent gradle check runs because the additional runs can help find flaky tests.