Add support for pinning package versions

[poettering]

I think it would make sense to optionally support "pinning" package versions, i.e. write the used RPM/DEB package versions to some file while doing an initial build, and if that file exists, use exactly these versions to build the package. The file could then be commited to git, ensuring that images are somehwat stabe in their package selection.

This should bring us a step closer towards reasonably reproducible builds.

I figure this means, at least for the Fedora case, to call directly into the python dnf module to retrieve these package lists.

[poettering]

Also see #748

[keszybz]

FWIW, just putting specific package versions in the manifest already works, to some extent. You can specify a full package nevra in Packages=, and dnf will honour that. I also use this to install locally built packages: when a path with slashes is given, dnf will install that package from disk.

We could retrieve the package lists, even post-factum by introspecting the rpm database or parsing the logs.

But the hard part is retrieving the specific versions of packages from somewhere. Packages in the distro stop being available from the mirrors after after a new version appears. It is possible to retrieve them with a bit of an effort (e.g. in Fedora's case, directly from koji). So pinning of a specific version would work if there is a local cache, or for a while as long as the distro doesn't update, but then it would break.

[Foxboron]

Does RH/Fedora have a package archive? For both Arch Linux and Debian all released packages are uploaded and stored on the archive. In the case of Arch, recreating any ISO from the past couple of years should be possible by just fetching them and passing them as arguments to pacstrap.

https://archive.archlinux.org/

How this is implemented in the current Arch Linux repro tooling; https://github.com/archlinux/archlinux-repro/blob/master/repro.in#L490

[behrmann]

No Fedora expert, but https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/ looks similar to the Debian archives. No idea though, whether this can be directly hooked into dnf (or yum for the more ancient releases there).

[DaanDeMeyer]

I'm not sure if this should be implemented by mkosi tbh. Users can either use --cacheonly to never hit the mirrors again after the initial download, which effectively pins the package versions. The other option is to setup your own mirror with the versions of the package that you want and then configure mkosi to use that mirror. That seems like it would scale much better than pinning individual package versions.

Closing this for now but we can reopen if needed.

[pbjhelmert]

I think both of the proposed alternatives are less than ideal due to their being hard to mesh with all of the nice semantics of Git, especially the change atomicity that git gives us, but also...

Users can either use --cacheonly to never hit the mirrors again after the initial download, which effectively pins the package versions.

Users would then have to ship around a tarball or similar of the cache, which seems less than ideal, especially if the other option of using mirrors is available.

The other option is to setup your own mirror with the versions of the package that you want and then configure mkosi to use that mirror.

I think a common pattern for environments which need these kinds of reproducible builds is to use some flavor of local caching proxy as a mirror where one can use a pinned set of packages from the mirror, potentially across multiple images, as well as use that same mirror for other systems running whatever distribution is being built. Should users have to create a mirror per image they need to lock down in this way?

In the case of RHEL variants, dnf's versionlock` plugin provides a lot of the pinning functionality that I'm after but that's kneecapped by the dnf plugin disabling, so an mkosi-native way to deal with this kind of requirement would be welcome.

[DaanDeMeyer]

I think both of the proposed alternatives are less than ideal due to their being hard to mesh with all of the nice semantics of Git, especially the change atomicity that git gives us, but also...

Users can either use --cacheonly to never hit the mirrors again after the initial download, which effectively pins the package versions.

Users would then have to ship around a tarball or similar of the cache, which seems less than ideal, especially if the other option of using mirrors is available.

The other option is to setup your own mirror with the versions of the package that you want and then configure mkosi to use that mirror.

I think a common pattern for environments which need these kinds of reproducible builds is to use some flavor of local caching proxy as a mirror where one can use a pinned set of packages from the mirror, potentially across multiple images, as well as use that same mirror for other systems running whatever distribution is being built. Should users have to create a mirror per image they need to lock down in this way?

In the case of RHEL variants, dnf's versionlock` plugin provides a lot of the pinning functionality that I'm after but that's kneecapped by the dnf plugin disabling, so an mkosi-native way to deal with this kind of requirement would be welcome.

For now we can look into re-enabling the versionlock plugin. Anything more than that is blocked by rpm-software-management/dnf5#833