Improve Test fixtures

[galvesribeiro]

Proposed change

We should adopt TestContainers to improve the test reliability in particular to spin up and down the servers.

Use case

Test Containers make it easy for us to manage instances of Nats to be used on unit tests and avoid manual management of processes. All that the test agent (CI machine, developer machine, etc) needs to have is the ability to run containers (i.e. Docker).

Contribution

I'll look on what it takes and write details later on here.

[mtmk]

this is good. thanks @galvesribeiro 💯 (to be fair I think someone else suggested before as well. can't remember who though) a few details:

• running the tests on windows is unstable - testcontainers would really help here
• current NatsServer.Start() method consuming way too much memory presumably log capturing etc added over time didn't help.
• there is a new class more lightweight under Platform.Windows.Tests which also might help
• current GitHub builds have a few well known flappers that can be looked into to improve the CI (I do this now and again)
• we also test against different server versions one being the main branch to help catch potential issues on server development. main branch builds won't be on docker hub I don't think.
• personal opinion: I've only seen containers used on the aspire project, I have almost zero experience, but if running the server can be 'abstracted' in some fashion.. maybe? maybe some kind of a switch/setting?
• we can maybe start by writing a few redundant tests using testcontainers? like a POC?

Just a brain dump 😅 I intrigued to see where we go from here 💯

[galvesribeiro]

we also test against different server versions one being the main branch to help catch potential issues on server development. main branch builds won't be on docker hub I don't think.

That shouldn't be a problem. We could build, as part of the CI, a local container image using main. Then run test containers from that local image. Not a problem at all. We can smuggle this into the actions. The other versions we pull from Docker hub as usual.

Also, we can have "experimental" builds that happens every time we merge something to main. Those could generate Docker images and push them to the container registry here on Github. But that ofc would need to be added to the server repository CI pipeline.

personal opinion: I've only seen containers used on the aspire project, I have almost zero experience, but if running the server can be 'abstracted' in some fashion.. maybe? maybe some kind of a switch/setting?

This will be trivial. All you would need is to have docker desktop installed on your machine (Mac/Windows/Linux) and it will be fine. The CI agent on GitHub Actions is actually a container which also support "Docker in Docker" meaning that we wouldn't have any issues running there as well.

we can maybe start by writing a few redundant tests using testcontainers? like a POC?

Yes, we can do it all separated in a different fixture and pick up the tests that are more likely to fail than others. Once it is safe, we just swap the fixtures on the rest.

[mtmk]

that sounds quite good @galvesribeiro! My last whinge if I may, sorry 😬 I think it's also good to be able to run the server on Windows and test againsts that as an option.

[galvesribeiro]

We can still use containers. Windows Containers is a thing, so we can still use the same API as long as we run the Windows tests on Windows agents. I never had to run Windows containers on GitHub Actions even tho I know there are Windows agents. I guess we will find out soon 💃🏻