QA effort for Cabal

[Kleidukos]

From what I could observe during the past couple of releases, it would seem that we would get a good amount of solid feedback if we systematise the use of manual quality assurance (QA) processes to supplement the test suite for the patches that have an impact on cabal-install.

Example: #8843

This is pretty trivial and certainly could have been caught much earlier.

Moreover, tests written by developers of a feature tend to be more vulnerable to the author's cognitive biases. However, if we ask of people the higher-level behaviour that should be checked later on by another human, we can:

• Find code paths that were not handled properly
• Use the new code paths as part of a larger workflow / usage, and not only in the context of unit tests.

Proposal

What I want to propose is twofold:

1. We add a section "QA Notes" in the PR template for people to fill when they are submitting changes to cabal-install
2. We make a conscious effort to build a team of volunteers who can give some of their time to test that the invariants / expected behaviours of a ticket hold well outside of the test suite.

This will also allow us to catch usability regressions earlier on, and this is a very good opportunity to onboard more people as this makes them feel involved in the development process.
There are real benefits to gain from this that go beyond strictly "testing" things.

I'd like to get the comments from the community of Cabal contributors on this.

[Mikolaj]

That's a great idea. Another example is tests that are hard to automate and/or hard to triage failures of when automated. E.g., the stuff I mentioned here: #8788 (comment). I bet there are many other kinds of checks that don't fit in CI, including those you mention, where problems are trivial to spot by a human in normal usage.

[jneira]

Well, I would say that we lack the very basis of unit and integration tests discipline.
Being Cabal a lib and cabal-install a cli tool those tests could cover a good amount of cases.
And we are merging lot of patches and features without tests or not enough ones.
Imho QA tests are necessary but without a solid integration and unit test suite they are defeated

[ffaf1]

It would be very useful, for all the reason stated above. I wonder what is the best way to onboard QAers (e.g. providing binaries or have them compile cabal?).

If we had a set of users running HEAD even for their personal projects only, that would pick up a lot of things testing does not pick up now.

[Kleidukos]

Yes I'm thinking of making them use HEAD and self-compile.

[hasufell]

This sounds like you want nightlies for cabal?

[Kleidukos]

@hasufell No, not because I disagree with the concept of nightly bindists but because I try very hard not to take suggestions like these as entries in my Christmas wish list. So, you say "nightlies for cabal", I hear "another CI sub-system to maintain upon which testing will depend". Maybe I'm wrong but I think we should start with self-compilation of HEAD.

[Kleidukos]

Alright, the effort has started, thank you to everyone who sent feedback, here and elsewhere. We had great input from @tamara-mandziuk about how it could be improved, and this has certainly given food for thought.