Solver Service

Status: WIP & Experimental

A standalone, OCurrent service for solving opam dependencies extracted and modified from OCaml-CI. This repostory contains:

• src/solver-service-api: The Capnp solver service API, the defines the format for sending requests and receiving responses from a service.
• src/solver-service: The core functionality of completing a solver request using opam-0install is contained here. There are binaries for running a solver-service which can communicate over stdin/stdout or over the network.
• src/solver-worker: An OCluster worker that can solve requests being submitted to OCluster as a custom job specification.

Example

A solver service

The ./examples directory contains a small CLI tool for testing the solver service over a TCP connection. It requires a package name and version to solve for (note it will use opam to fetch the opam file information). To test it, you first must install and run the solver service. It spawns workers by recursively calling itself (using Sys.argv.(0)) so it is important to run it using its proper name rather than with dune exec --.

$ solver-service --address=tcp:127.0.0.1:7000
Solver service running at: <capnp-address>

Copy the <capnp-address> and run the example binary passing in the address.

$ dune exec -- ./examples/main.exe --package=yaml --version=3.0.0 <capnp-address>

A solver worker

The solver worker can join an existing pool on a scheduler and solve request jobs sent as a custom job type. The example submit.ml will send such a job to a scheduler (you need to supply the submission cap file).

To try this example locally, first get the scheduler up and running.

$ mkdir capnp-secrets
$ ocluster-scheduler --capnp-secret-key-file=capnp-secrets/key.cap --capnp-listen-address=tcp:127.0.0.1:9000 --pools=solver --state-dir=var --verbosity=info

This will write an admin.cap file into capnp-secrets. We can use this to add a new client and get a submission capability file.

$ ocluster-admin --connect ./capnp-secrets/admin.cap add-client solver > capnp-secrets/submission.cap

Now, we need to connect our solver worker to the pool. In a new terminal connect the worker using the pool registration cap file.

$ dune exec -- solver-worker --connect=capnp-secrets/pool-solver.cap --name=solver-1 --state-dir=var --verbosity=info

With this running we can use the example submission pipeline to solve the dependencies for the obuilder repository on Github.

$ dune exec -- examples/submit.exe --submission-service=capnp-secrets/submission.cap

You should then be able to watch the pipeline in action at http://localhost:8080.

A solver worker by docker-compose

Start a solver-worker connected to a scheduler.

$ docker-compose -f docker-compose.yml up

To reach the scheduler, we need to add its hostname in /etc/hosts file.

$ cat /etc/hosts
127.0.0.1   localhost
127.0.0.1   scheduler

Get the Mountpoint by inspecting the capnp-secrets volume.

$ docker volume inspect solver-service_capnp-secrets
[
    {
        "CreatedAt": "2023-06-02T17:00:23+02:00",
        "Driver": "local",
        "Labels": {
            "com.docker.compose.project": "solver-service",
            "com.docker.compose.version": "1.29.2",
            "com.docker.compose.volume": "capnp-secrets"
        },
        "Mountpoint": "/var/lib/docker/volumes/solver-service_capnp-secrets/_data",
        "Name": "solver-service_capnp-secrets",
        "Options": null,
        "Scope": "local"
    }
]

Be in root to get the admin.cap file form Mountpoint path. In macOs the command is sudo su.

$ mkdir capnp-secrets
$ sudo -i

# cd /var/lib/docker/volumes/solver-service_capnp-secrets/_data
# cat admin.cap
capnp://sha-256:rMn7cNJxSE...
# exit

$ echo "capnp://sha-256:rMn7cNJxSE..." > capnp-secrets/admin.cap

The scheduler and a solver worker is already started, we can now, get the submission.cap file. W could only use it for submitting solve builds.

$ ocluster-admin --connect capnp-secrets/admin.cap add-client solver > capnp-secrets/submission.cap

We can use submission.cap file to submit jobs using examples/submit.exe or use ocaml-ci-service by passing the file via --submission-solver-service.