Create self-describing bottles

[justenstall]

Verification

• This issue's title and/or description do not reference a single formula e.g. brew install wget. If they do, open an issue at https://github.com/Homebrew/homebrew-core/issues/new/choose instead.

Provide a detailed description of the proposed feature

Update the bottle artifacts pushed to the homebrew/core container registry so they are self-describing. Currently the bottle artifacts do not contain the formula's API metadata.

This can be accomplished by uploading the formula's API JSON data and referencing the blob as the bottle manifests' config.

What is the motivation for the feature?

Combined with #17837, this would allow bottles to be installed without the API.

Two sets of metadata exist for a bottle, the tab contained in an annotation and the API data. The tab is a subset of the necessary information and could contain conflicting information.

How will the feature be relevant to at least 90% of Homebrew users?

Faster install times without downloading the full formula index from the API.

What alternatives to the feature have been considered?

Not implementing the feature.

[Bo98]

On the surface, this seems reasonable and useful from at least a mirroring point of view (as it's a bit tricky to mirror currently).

However it is a bit more complicated. The data in the API allows many Homebrew commands to work offline and significantly faster, such as brew deps, brew info (--json), etc. As more concrete examples: the requirements list is useful to fail fast if something isn't supported on your OS, and dependency tree is ideal to know ahead of time as we fetch everything before parsing.

There's also a security element. We sign the API JSON so it cannot be tampered with by mirrors. No independent protection exists for manifests (it's covered by the bottle sha256s being included in the signed API JSON).

I would like to lean on the manifests more overall, but only if it doesn't degrade the above.

[MikeMcQuaid]

There's also a security element. We sign the API JSON so it cannot be tampered with by mirrors. No independent protection exists for manifests (it's covered by the bottle sha256s being included in the signed API JSON).

This is my main concern here, too.

[justenstall]

However it is a bit more complicated. The data in the API allows many Homebrew commands to work offline and significantly faster, such as brew deps, brew info (--json), etc. As more concrete examples: the requirements list is useful to fail fast if something isn't supported on your OS, and dependency tree is ideal to know ahead of time as we fetch everything before parsing.

I'm not proposing getting rid of the API or its data, I am proposing that the exact same data should be stored alongside the bottle in the container registry.

There's also a security element. We sign the API JSON so it cannot be tampered with by mirrors. No independent protection exists for manifests (it's covered by the bottle sha256s being included in the signed API JSON).

OCI artifacts can be signed and verified in a similar way to how Homebrew signs the API JSON, Notary and cosign are both established standards for signing and verifying OCI artifacts.

[MikeMcQuaid]

I'm not proposing getting rid of the API or its data, I am proposing that the exact same data should be stored alongside the bottle in the container registry.

How would a client (e.g. Hops) know what formulae are available without using the API here?

Notary and cosign are both established standards for signing and verifying OCI artifacts.

TIL, thanks.

I think that signing would be a hard requirement on our end for either this or #17837. It might be the best first step, here.

[woodruffw]

OCI artifacts can be signed and verified in a similar way to how Homebrew signs the API JSON, Notary and cosign are both established standards for signing and verifying OCI artifacts.

JFYI: We don't do it at the OCI layer, but Homebrew does indeed use Sigstore (the stack under cosign) to attest to all of its (core) bottles: https://github.com/Homebrew/homebrew-core/attestations.

(Not sure if this is relevant to you; I just noticed the cosign reference 🙂)

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.