Feature: install from manifest

[passcod]

Make a new manifest that specifies a list of crates/tools to be installed. The intent is both to keep a system/user-wide list of tools with versions for local use, and also a list of project-specific tools in a format that can be checked into version control. There should also be a lockfile, for the same mechanism as Cargo deps: version requirements in the manifest, and exact versions in the lockfile.

• The lockfile could also include URLs and checksums, providing a measure of security even without signing.
• It would be nice to have a set of tools like cargo add, cargo rm to manipulate the manifest.
• Needing to manipulate the lockfile points to a wider set of commands than the simple cargo binstall interface can accommodate.
• We could embed the manifest in the Cargo.toml, or have our own file. We'll definitely need our own file as a lockfile, so it could make more sense to have our own manifest file.
• Out of scope: non-crate tools. E.g. installing valgrind is not our concern.

I think this would be highly valued, and generally be very useful, not just for cargo projects.

---

I propose (and am working on) the following:

• A new tool under this project umbrella, cargo tools (a cargo-tools crate exists, but isn't used).
• A new manifest file, Tools.kdl, and its NLJSON lockfile, Tools.lock
• Default command cargo tools would print the status (what's installed, what isn't, what can be updated, what can be removed) and an interactive prompt to do something about it (choices: install as lockfile, update compatible, update breaking, do nothing). That way 90% of the time all you want to do is run cargo tools.
• A suite of subcommands: cargo tools add, cargo tools rm, cargo tools install, cargo tools update, cargo tools init, cargo tools status, etc

Manifest:

.version 1 // not required

.defaults {
    // Where to install binaries, defaults to `./bin` for projects and to `$HOME/.cargo/bin` for global
    install-into "./vendor"
}

// with version requirement
cargo-binstall version="0.9.1"

// when just any version will do
cargo-watch

// override defaults for this particular tool
watchexec {
    install-into "./bin"
}

A typical manifest could look like:

cargo-binstall
cargo-watch
cargo-make
cargo-edit
cargo-deb version="1.39.1"
cargo-generate-rpm version="0.8.0"

Sample of lockfile (pretty-printed for readability, each top level object would be its own line):

{"version":1}
{"req":{
  "name":"cargo-binstall",
  "version":"0.9.1",
  "metadata":"crates.io",
  "source":"repository",
  "template":{"pkg-fmt":"..."},
  "install-into":"./vendor"
},"pkg":[{
  "version":"0.9.1",
  "target":"x86_64-unknown-linux-musl",
  "source":"https://github.com/ryankurte/cargo-binstall/releases/v0.9.1/download/cargo-binstall-x86_64-unknown-linux-musl.tgz",
  "checksums":{"b3":"c5cd314cea5aca5391427bf9885d11f1fbc86d79679353a36d1b89480422149d"},
  "bins":["cargo-binstall"]
}]}

It's not practical to look for and download all available targets, so only the ones that are installed by the host / runner are written to the lockfile. When a user on a different host target runs cargo install, new files are downloaded and added to the lockfile; this prints a warning/notice to the console. If running with --locked, this causes an error instead. --keep-version is the middle ground, allowing versions to be kept exactly the same as locked, but new files to be downloaded if their target is not in the lockfile already.

[NobodyXu]

Perhaps we can add that as a subcommand and make cargo-binstall a multi-call binary?

[passcod]

Are you concerned about size? I was just thinking of having two binaries.

[NobodyXu]

Are you concerned about size?

Yes.

I don't think we can simply invoke cargo-binstall in cargo-tools since we need to check for the latest release.

[passcod]

I don't mind a multi-call, but on windows we'll still need to distribute two (identical) binaries, I think. Probably the best we can do without going down the path of shipping a dynlib along. Also not entirely sure how multicall binaries work in the cargo context, especially with the cargo install cargo-binstall pathway?

[NobodyXu]

Also not entirely sure how multicall binaries work in the cargo context, especially with the cargo install cargo-binstall pathway?

It seems that the user would have to create the symlink themselves in these cases.

cargo-install does not support post installation script.

[passcod]

Right, well, that's not a great experience. I wonder if we could rig it such that it defaults to two binaries, but produces a multicall given a feature. That way we can generate prebuilds with the smaller multicall setup, and cargo install still works.

[NobodyXu]

Right, well, that's not a great experience. I wonder if we could rig it such that it defaults to two binaries, but produces a multicall given a feature. That way we can generate prebuilds with the smaller multicall setup, and cargo install still works.

I think we can create bins/cargo-tools.rs and bins/cargo-binstall.rs which simply call the multicall main.

Since the binary is named cargo-tools, multicall would resolve to call the right "main".
Same for cargo-binstall.

[ryankurte]

ooh, i like the idea of this! would it be too self important to call this Tools.toml instead of CargoTools.toml?

[ryankurte]

hey thinking about this a bit more, how do you manage multiple projects with different tool requirements? it feels like a global manifest is a different problem to a per-project manifest (whether that's at a crate or workspace level), particularly given projects could have different tool version requirements.

if you want a -project- to be able to specify required tools it might be plausible to add a tag in Cargo.toml then use the existing build-dependencies and lockfile... as well as necessary to locate the binaries -within- the project / get that on the path somehow?

[passcod]

Yeah, the way I was thinking is that in project mode, we install to a directory inside the project, so it's completely separate from the global installs. There's no convention in rust to have a project-local bin folder that the tooling uses like in Node (node_modules/.bin), which makes this a bit unfortunate to use (vendor/tool args rather than tool args, for example), but plenty of projects have a "scripts" folder, which this would just be an extension of, I guess.

[passcod]

There's also the cargo xtask pattern, but I'm not sure if we could reasonably use that or hook into it for anything.

[passcod]

Implemented by (third party project) cargo-run-bin, see #1514.