github: Use Canonical runners for scheduled system tests

[roosterfish]

This PR splits the rather complex matrix system test into three groups.
The system test code is moved into a repo local action which is leveraged by each of the groups.
See an example of the restructured tests workflow here.

In addition the "instances" suite is now executed also on the Canonical runners when the workflow is triggered by schedule.

[roosterfish]

@masnax I did some tests regarding this error.

Unfortunately the timeout set in the MicroCeph GetConfig client func is set to only 5s.
Based on your suggestion in the meeting earlier, can it be that right after forming the MicroCloud, the proxy is waiting for something in the cluster to settle before it can forward the request to MicroCeph's local unix socket?

I have bootstrapped a single node MicroCloud and fired requests to /1.0/services/microceph in parallel.
Right around when the MicroCloud is bootstrapped I saw a delay in the response which could proof that something is going on there.

[masnax]

I have bootstrapped a single node MicroCloud and fired requests to /1.0/services/microceph in parallel. Right around when the MicroCloud is bootstrapped I saw a delay in the response which could proof that something is going on there.

Well this current failure is happening long before MicroCloud is bootstrapped, as it happens right after system discovery and before asking any setup questions. In the bootstrap case, the only delay would be related to refreshing the truststore and waiting for the lock, but even that wouldn't happen on a single-node request as it all goes through the unix socket which skips truststore verification.

When bootstrapping, the listeners also restart, so that could be the delay you're seeing locally. But again that wouldn't affect the test failure since it's not during bootstrap.

can it be that right after forming the MicroCloud, the proxy is waiting for something in the cluster to settle before it can forward the request to MicroCeph's local unix socket?

this is the whole local proxy block in MicroCloud so it's definitely not waiting for anything here.

Since it's a network request, there is the additional overhead of authHandlerMTLS pulling the truststore.

[roosterfish]

Well this current failure is happening long before MicroCloud is bootstrapped, as it happens right after system discovery and before asking any setup questions

Mh it looks we can fix it by waiting for microceph cluster bootstrap to settle and only continue if it's done.
I have added another commit that adds a wrapper function we can use throughout the test suite to wait until microceph status reports the single node cluster services are present. I found waiting for this condition looks to be enough.

[masnax]

Well this current failure is happening long before MicroCloud is bootstrapped, as it happens right after system discovery and before asking any setup questions

Mh it looks we can fix it by waiting for microceph cluster bootstrap to settle and only continue if it's done. I have added another commit that adds a wrapper function we can use throughout the test suite to wait until microceph status reports the single node cluster services are present. I found waiting for this condition looks to be enough.

Is this something that can be checked over the API? Perhaps microceph cluster bootsrap shouldn't return until all its services are finished, or have a ready API that we can check against before sending requests.

[roosterfish]

Is this something that can be checked over the API? Perhaps microceph cluster bootsrap shouldn't return until all its services are finished

Issue logged in canonical/microceph#473.

[roosterfish]

@MggMuggins have you ever seen this one https://github.com/canonical/microcloud/actions/runs/12031420307/job/33588930317?pr=469#step:17:1490?

It might be that this is caused because the runners are slower. We already saw various others scenarios caused by the "slowness". Maybe you have an idea.

[MggMuggins]

IIRC the trust store is held in memory in MicroCluster and synchronized (fanotify?); I think the recovery process doesn't use the in-memory synchronization because it's expected that it won't be accessing the trust store at the same time as any other thread. I wonder if this was an incorrect assumption. I'll try and take a look this afternoon.

[MggMuggins]

I have downloaded the logs for the failed PR run and will plan to look into this next pulse.

[tomponline]

needs a rebase please

[roosterfish]

@tomponline @masnax rebased.