Reacting to state changes of health checks?

[nscuro]

Hey all,

we're running a few Quarkus applications that are both connected to a database, and process Kafka messages.

Processing of the messages involves interacting with the DB. So if the DB is unavailable, all processing will inevitably fail.

Both the DB connection pool and the Kafka consumer have independent health checks.

What we'd like to achieve is to pause the Kafka consumer when the database health check starts failing, and resume it again once the database health check succeeds again. Consumers do have failure handling and retries, but we would like to avoid producing errors in the first place.

Ideally, each health check would publish an event to an internal event bus when its state changes, so other components of the app get an opportunity to gracefully pause / resume their work.

I think this can be achieved with custom health checks and vert.x event bus, but it won't work for the default health checks (e.g. DataSourceHealthCheck).

Anyone have ideas how to best go about this? Are there any best practices around this kind of thing?

[quarkus-bot[bot]]

/cc @jmartisk (health), @xstefank (health)

[xstefank]

Hi @nscuro. Not sure I understand what you're after. But all HCs are CDI beans. So is this what you mean?

package org.acme;

import io.quarkus.agroal.runtime.health.DataSourceHealthCheck;
import io.smallrye.health.api.HealthGroup;
import org.eclipse.microprofile.health.HealthCheckResponse;

@HealthGroup("testHC")
public class ExtendedDBHC extends DataSourceHealthCheck {

    @Override
    public HealthCheckResponse call() {
        HealthCheckResponse result = super.call();
        if (result.getStatus().equals(HealthCheckResponse.Status.DOWN)) {
            System.out.println("Send message here");
        }
        return result;
    }
}

@HealthGroup("testHC") because I don't want it to fight with the original DataSourceHealthCheck. But you can also set quarkus.datasource.health.enabled=false and then you can make ExtendedDBHC also @Readiness as the original DataSourceHealthCheck.

[xstefank]

Also pushed my test if you want to try it yourself - https://github.com/xstefank/quarkus-reproducers/tree/main/hc-messages

[nscuro]

Thanks a bunch @xstefank, very helpful! That looks exactly like what I was looking for. :)