Add CI tests?

[tlambert03]

Hey @marktsuchida,

As discussed in #372, there isn't much unit testing set up in this repo, and what has been written isn't getting run on CI. I'm willing to put in a little work to get that setup, but wanted to discuss strategy first.

I know there's a little logic in place in the micro-manager repo to test stuff here, but it looks like that relies on manual local testing right? (no CI setup there either).

Being a C++ repo, the obvious choice is to continue with googletest here, but one other option would be to go ahead and actually build pymmcore around this repo and run pytest on it. The advantage would be that it would be a very full integration test for at least one half of the downstream use cases, and would enable people to add new tests very easily (i.e. without having to learn googletest). the downside of course is that it's a bit weird to test a c++ library with python or java (even if that library is usually used by one of those languages).

Anyway, just curious to get your thoughts before putting in any work. If you had time to devote to setting up more testing here, how would you do it?

[marktsuchida]

Much to discuss here, but a few initial thoughts:

• We do have an automated build using Jenkins, albeit post-merge (tip of main is built). Although the existing tests aren't currently run there, they could be (for the macOS build, at least).
• The existing unit tests have only ever been run on macOS and Linux -- I don't think I have ever tried to set them up on Windows. If I were to improve these, I would first switch from the antiquated GoogleTest (which was the best option pre-C++11) to Catch2, the modern favorite, taking advantage of the fact that we have so few tests. Probably more efficient to do this after unifying the build system (to use Meson on all platforms), but not impossible to do now.
• I'm sure the situation is chicken-and-egg -- we would likely be adding more unit tests if it were easier to do so (I know I would).
• I think it makes a lot of sense to have integration tests, orthogonally to whatever we do (or don't) with the unit tests. I know you already have such tests, and it makes sense to take advantage of those (or equivalents tailored to bare pymmcore) to detect incompatible changes (or outright bugs) early.
• I think it is quite feasible to build pymmcore and just a few device adapters (probably DemoCamera and SequenceTester; also Utilities so that it can be tested) on GitHub Actions. It probably makes sense for pymmcore to be capable of building what's needed (would also address Ship DemoCamera with pymmcore pymmcore#19) and to contain the test cases, but they can be run on mmCoreAndDevices PRs (I assume this is what you meant?).
• For the last part (building a few device adapters in pymmcore's build system), it probably makes sense to go ahead and switch pymmcore's build system to Meson, which is a C++ build system that can also act as a Python build backend (in place of setuptools). I've done some preparation for this, and wouldn't mind working on it. Building device adapters (which are neither Python modules nor shared libraries, but shared modules) in setuptools is probably not worth the bother.
• If extending the integration tests also to real device adapters, I'm not so sure it would make sense for the tests to be in pymmcore. But probably the top priority is to get MMCore and MMDevice under some form of testing; the rest we can discuss later.

[tlambert03]

perfect. all makes sense and very helpful. waiting for the move to meson makes a lot of sense. I recall seeing that a year ago or so and I think i even played with that branch a bit.

Does this branch still reflect the state of the meson transition? https://github.com/micro-manager/mmCoreAndDevices/tree/meson

Is this something that's kind of on the back-burner for now (if for no other reason that limited time/resources)? or is it something you're actively working on?

linking #86 for related discussion as well.

[marktsuchida]

Nope, that branch is older than the current plan (micro-manager/micro-manager#1392) and the overall structure is meant to be quite different. This has indeed been on the back-burner for a while, but I intend to get back to it before the end of the year. But pymmcore can use Meson independently of this repo, because pymmcore doesn't use any of the build scripts here anyway (for now; once MMDevice/MMCore build with Meson and live in their own repos we will be able to make them proper dependencies, but that can happen later). So we can get started on that first and avoid having the integration tests you propose be coupled to the lager build system project.