Support custom jobs

[patricoferris]

This PR adds custom job specifications to the cluster API using the Cap'n'p AnyPointer. A user can then convert their payload to a pointer and have their worker handle their custom cases. The kind field was meant to allow distinguishing between multiple custom job specifications (I don't know if there's a better way to do that).

The simple_custom test is meant to pretend the OBuilder job specification doesn't exist and creates a custom job where the payload is an OBuilder struct with a spec field.

The PR also exposes the default_build in the cluster-worker library. The idea there is that if #151 is merged and released then people can extend the worker to custom jobs whilst still supporting docker and obuilder ones? It might be nice to then expose some of the logic currently in bin/worker too. See this branch of solver-service for an example of using the cluster_worker libraries.

[talex5]

Seems reasonable.

(I doubt it matters, but I'm slightly disappointed that serialising a custom type requires doing it first to a temporary area and they copying it to the final message, instead of writing it in place. I guess the alternative is registering user-provided conversion functions and using an open type or something.)

[patricoferris]

Thanks for the review. I also hit that snag and opted for the more wasteful approach of copying (mainly because I hadn't even considered open types so my use of conversion functions didn't work!).

[patricoferris]

This PR now also adds support for submitting the custom jobs using the ocurrent ocluster plugin. In order to build a "solver worker" it would be great to make the cluster worker library publicly available. Here I've just added it to the ocluster package with ocluster.worker but that's probably not desirable (it was just to unblock solver-worker) in which case it will need #151 probably.

[talex5]

Thanks! Needs rebasing, with the new level attribute added.

Not sure about exposing the worker - if we want people to build on our worker then we need to turn it into a proper library. Most workers won't want to be testing for the docker daemon, reporting docker metrics, etc. Might be better to drop that bit for now.

[patricoferris]

Thanks for the review @talex5 (and sorry for the delay)!

The PR is rebased, the worker is no longer public, custom jobs require a cache_hint and the Custom API for Current_ocluster now also has a test.

[talex5]

Looks good, except I think we need to do something about the OCurrent cache key.

[talex5]

I don't think this will work. When the payload changes, we'll need a new database key for it.

So I think we need some kind of Custom.key function to extract the important parts. We only need that on the sending side, so it might make sense to split action into two types, one for sending and one for receiving. I had a go at that here (just the splitting; not getting the key):

talex5@4aa07b2

Also, this cache key is only needed for the OCurrent plugin, so we might want to make it optional for other users of OCluster.

[patricoferris]

Ah good point, that makes sense. I'll see if I can extend your commit with the key part. Thanks :)) !

[talex5]

Thinking about it more, perhaps custom key generation should be part of the Current_ocluster plugin's configuration?

Do we need OCurrent support for custom jobs at the moment? If not, that could just be taken out and added later in a separate PR.

[patricoferris]

Thinking about it more, perhaps custom key generation should be part of the Current_ocluster plugin's configuration?

How would this function get to the Key module to change how custom jobs are digested? Would we provide a functorised version that needs a custom key generating function with the current implementation being the default one provided, or perhaps I'm missing something ?

Do we need OCurrent support for custom jobs at the moment? If not, that could just be taken out and added later in a separate PR.

It would definitely be useful to have OCurrent support for custom jobs, in the context of the solver service we'd want to be able to submit jobs in a pipeline for doing the solves, but as you said maybe it is easier to do this in a follow-up PR.

[talex5]

How would this function get to the Key module to change how custom jobs are digested? Would we provide a functorised version ...

That sounds reasonable to me. If that turned out to be awkward, another option would be adding the function to the Key.t when the key is created (the plugin configuration is available then). A third option is having some kind of registry of handlers for custom jobs.

in the context of the solver service we'd want to be able to submit jobs in a pipeline for doing the solves

In the case of the existing CIs, there is a pre-existing "analysis" build step that submits jobs to the solver internally, so that wouldn't need to use the plugin directly. In fact, even the existing cluster build jobs don't use the Current_cluster builder directly, but instead define their own builder using the low-level Current_ocluster.Connection API.

[patricoferris]

Thanks @talex5 for the pointers, I didn't know about the Connection API! I think all of the custom parts have been removed from Current_ocluster and the tests now show how to use the underlying Connection.t to submit custom jobs (I think...)

[talex5]

Looks good - thanks!