Testing and Code coverage

[PeachesMLG]

I think we should try to move towards automated testing in all our repos I suggest

Jellyfin-Web
Unit Tests - Jest (https://jestjs.io/docs/tutorial-react)
End to End Tests - Cypress (https://www.cypress.io/)

Jellyfin
Unit Tests - Xunit (already somewhat implemented)

I would also suggest in order to ensure a level of automated testing, we should implement some code coverage tools.
There are some tools out there which can measure code coverage and code quality which we should be taking advantage of.

I've used Sonarqube (https://www.sonarsource.com/open-source-editions/sonarqube-community-edition/) before and have found it useful.

I havent used any other tools, but there are a lot out there which can probabbly measure code coverage (aswell as general code quality) https://stackoverflow.com/questions/38635/what-static-analysis-tools-are-available-for-c

Why implement this:
Adding automated tests will help us identify breaking changes in our code and reduce the effort of manually testing each release, in an ideal world we will have 0 manual testing, and rely fully on CI/CD which will enable us to push releases out quicker.

Measuring the code coverage will then tell us where the gaps our in our unit tests and ensure that every pr will also add the necessary tests for the newly added code

[PeachesMLG]

For anyone who hasnt used cypress before, I setup a quick example here: https://github.com/PeachesMLG/cypress-example

You can see the cypress tests here: https://github.com/PeachesMLG/cypress-example/blob/master/cypress/e2e/website.cy.ts

And the results of the github actions here: https://github.com/PeachesMLG/cypress-example/actions/runs/8215043299

[gnattu]

A significant issue for end-to-end testing is that it requires tremendous manpower to achieve meaningful coverage, especially for complex projects like Jellyfin-web. When my company added something similar, the entire QA team and the development team paused all work for a month just to write these test cases. This would be a very, very challenging task for us to replicate as we our team size is much smaller.

Another problem is the machines that run those end-to-end tests. Jellyfin is not a simple web project that any browser can run the tests fine. It requires a browser with actual GPU and media capability to perform a lot of tasks, and that would also be challenging as the GitHub runners do not provide this kind of machine, at least not for free.

Unit testing would be much easier to implement, but achieving great coverage still requires significant manpower. However, it is more realistic than fully automated end-to-end testing.

[PeachesMLG]

Yeah, obviously if we try to add tests for everything all at once it would take a while. but we can definatly start incrementally adding tests over the next couple months in different areas.

By no means am I suggesting we pause everything and get 100% code coverage all in one go

[ferferga]

Isn't Playwright the default go-to solution as of today for E2E?

[PeachesMLG]

Cypress is very big and widely used, its also the one I have most knowledge with hence why I suggested it. Ive never used playwright before though, are there any features playwright provides which cypress doesnt?

[nielsvanvelzen]

We're already using Vitest in other repositories (cast receiver / typescript sdk) so I'd argue we should use that instead of Jest.

[PeachesMLG]

Ive never used that before, but it looks like it has integration into react components, so yeah if we are already using that in places I would agree

[thornbill]

Vitest was also recently introduced in web 😉

jellyfin/jellyfin-web#4829

[PeachesMLG]

A significant issue for end-to-end testing is that it requires tremendous manpower to achieve meaningful coverage, especially for complex projects like Jellyfin-web. When my company added something similar, the entire QA team and the development team paused all work for a month just to write these test cases. This would be a very, very challenging task for us to replicate as we our team size is much smaller.

Another problem is the machines that run those end-to-end tests. Jellyfin is not a simple web project that any browser can run the tests fine. It requires a browser with actual GPU and media capability to perform a lot of tasks, and that would also be challenging as the GitHub runners do not provide this kind of machine, at least not for free.

Unit testing would be much easier to implement, but achieving great coverage still requires significant manpower. However, it is more realistic than fully automated end-to-end testing.

So for end to end testing, we have two options. the first one is like you said, we can run both the frontend and backend and do full end to end tests.

The other option is we can mock the backend with fake api's and websocket connections.

Both of these come with their own positives and negitives

[PeachesMLG]

If the compute power of github actions is a concern, we can also look into something like https://buildkite.com/ which can allow us to run code on our own infrastructure, the drawback obviously being that this would require us to use our own infra.
It would also add more complexity since we would now have azure (which we are moving away from) gh actions, and buildkite

[thornbill]

As I mentioned in another comment we recently added vitest in web… but there is a lot of work needed to have any significant amount of coverage… and the legacy code is essentially untestable.

I had started an E2E test project that uses webdriver.io. The interesting thing about the webdriver project is that it could also support tests for all of our apps. https://github.com/thornbill/jellyfin-test-suite

[PeachesMLG]

Sure, but there are still some smaller websites we can add it to, for example I see no reason why https://github.com/jellyfin/jellyfin.org cant have any testing

And yes it will take a while to add 100% coverage, but we can still start with bits at a time