ENH - Improvements to automated accessibility tests

[trallard]

As we have been discussing some accessibility work for the theme, we have also discussed the need to develop a sustainable semi-automated way to audit for accessibility issues (and improvements, of course) and define what we mean by accessible -> or more, how this theme is WCAG conformant or not.

In that spirit, I am opening this issue to start moving the needle in this direction.

What has been done so far

1. Automated a11y testing and reporting was introduced by Add pa11y testing and reporting #294 and later reverted in Remove pa11y infrastructure from this documentation #751
2. Add lighthouse in MAINT: New action for lighthouse #572 through GH actions

There are several limitations and improvements to be made to the current approach:

1. The Lighthouse results are not very intuitive or findable.

   1. For example for this workflow run, one can access the results either by downloading the zip from the artefacts or clicking on the upload URLs from the workflow logs. Though at first glance, it is not evident that there is a report for each page being tested (I missed this the first time, and the HTML report names are not helpful either).
   2. Also, I am not sure if the current tests are being run on the light or dark version of the theme (or both), and I could not find a way to determine this (I am assuming it is the light one as the default theme?)

2. A few issues are encountered and reported (unfortunately, any automated tool will only catch up to 30-40% of the accessibility issues - so we can only aim to increase our coverage here and complement it with manual testing/verification). For example, for the workflow run mentioned above, the tests report a 90 score (a weighted average of all the accessibility audits) for the admonitions page.

   

   For example, all the colour contrast issues identified in Accessibility Colour Contrast Issues #1052 are missed altogether.

3. The only mention to the accessibility scripts I found in the docs is in the Accessibility section for users. Still, I could not find any other mentions for contributors or maintainers on how to find these results or improve the accessibility tests.

Suggested improvements

The road to ensuring the theme is WCAG conformant will be iterative and continuous, as discussed in other issues. So our testing strategy should also be iterative.
In the near future, I propose the following steps/improvements:

1. Add a minimum set of automated accessibility tests with Playwright:
   1. We can use axe-core which is the same package used by other tools like pa11y and even Lighthouse
   2. We are already using Playwright for JupyterLab accessibility tests, and there is also an issue here, Test our JavaScript with Playwright #585 discussing using Playwright for testing the theme, so it seems like a good direction
   3. To start with, we can run the tests on the same pages that the Lighthouse audit is doing (Kitchen sink sections)
2. Document these tests, what they are auditing (and what not) and the such to ensure y'all and other contributors have this at hand.

I am proposing starting with a minimal set of automated accessibility testing through Playwright and GitHub actions for now.
I want to make sure we are not adding more maintenance overhead and complexity for you. So once we have this MVP we can then discuss/brainstorm if we need better reporting of the WCAG violations encountered or if we can and should extend the tests and make these more robust or suitable as we continue other remediation work.

@gabalafou has been working on adding the automated tests for JupyterLab, so we will be working together on this for the PyData theme.

As usual, if you have any questions or else ping me and I will get back to you

[drammock]

+100! Thanks for sticking a plan and keeping us moving in the right direction