Add experiment requirements

[ecstatic-morse]

Resolves #430.

This adds a requirement field to Experiment, which specifies what requirements (if any) an agent must meet to run that experiment. Requirements are not structured, they are simply strings which are compared for equality. Agents advertise what requirements they meet via the request body when getting their configuration from the /config endpoint. Only experiments whose requirements are a subset of an agent's capabilities can be assigned to that agent. By default, experiments created via the CLI have no requirements--they can be run on any machine--and ones created via webhook (e.g. from craterbot) have the linuxrequirement--they must be run on a Linux machine.

For now, an experiment can have zero or one requirements. In the future, we may want to allow experiments to have an arbitrary number of requirements, and only agents which match all of them will be selected to run that experiment. This would require an additional table however.

For a crater run to be scheduled on both a Linux and Windows agent, you'll need to invoke craterbot twice, once with the linux requirement and once with the windows requirement. Perhaps later we could add some sugar for this?

r? @pietroalbini

[pietroalbini]

Left a few things to improve but otherwise the changes in this PR look great! In addition to the comments I left it would be nice if the latest agent capabilities were recorded in the agents table and displayed in the agents page of the UI.

[ecstatic-morse]

If we want to persist agent capabilities to the database, maybe the agent should post its capabilities at start-up when doing GET /config. Do you foresee capabilities changing frequently enough that we need to repost them on every call to /next-experiment?

[pietroalbini]

If we want to persist agent capabilities to the database, maybe the agent should post its capabilities at start-up when doing GET /config. Do you foresee capabilities changing frequently enough that we need to repost them on every call to /next-experiment?

Yeah sure, you can post them on GET /config.

[ecstatic-morse]

Okay, I add a normalized agent_capabilities table, and use it in a subquery when fetching the next experiment. This required a pretty major rewrite of the PR, so you should take another look.

With the exception of including agent capabilities in their HTML status page, all of your previous concerns should be resolved. Capabilities is now a proper data structure (albeit one that has deref coercion to enable extend and into_iter), and both the define-ex and edit-ex commands allow setting the requirement of an experiment.

I changed GET /config to POST /config, since now behavior depends on the request body. I'm not sure how zealotic you are about REST semantics 😄. I left in the route for GET /config, but it always uses a default set of capabilities (["linux"]).

Also, I'm not quite sure how to handle requirements for Assignee::CLI. Right now, fetching the next experiment for a CLI will ignore experiment requirements. I left a FIXME in src/experiments.rs to this effect.

[pietroalbini]

All the current code looks good! Once the capabilities are added to the HTML page I think we're ready to merge this!

[pietroalbini]

I'm not sure how zealotic you are about REST semantics.

Not that much :D

Also, I'm not quite sure how to handle requirements for Assignee::CLI. Right now, fetching the next experiment for a CLI will ignore experiment requirements. I left a FIXME in src/experiments.rs to this effect.

Having the CLI ignoring requirements is fine, as requirements don't really make sense in a CLI environment.

[ecstatic-morse]

@pietroalbini Capabilities are now rendered to the status page.

I based Agent::with_capabilities in the second to last commit on Agent::with_experiment (immediately above the newly added function). Maybe there's a better way to do lazy initialization here?

[pietroalbini]

Looks great! Other than the minor nit this is ready to be merged.

[pietroalbini]

Thanks ❤️

@bors r+

[bors]

📌 Commit d54cd7f has been approved by pietroalbini

[bors]

⌛ Testing commit d54cd7f with merge 1ed788c...

[bors]

☀️ Test successful - checks-travis, status-appveyor
Approved by: pietroalbini
Pushing 1ed788c to master...