core: Filter locked packages by checksums before depsolving #1927
Conversation
Don't just filter down packages by NEVRA, but also filter out those that don't match the checksum too. We were enforcing checksum matches already before this, but only *after* depsolving and simply erroring out if they didn't match. However, because of how RPM signing is implemented in Fedora, it is possible to have the same NEVRA in two different repos, each with two different hashes. E.g. right now for example, `efivar-libs` wasn't rebuilt for f31, and so f31 is just shipping the f30 RPM, but signed with the f31 key. And of course, we also had the f30 version in the pool. This patch allows us to transition over to the f31 version with everything else by not getting thrown off by the f30 version already in the pool. (Still need to investigate how the pool will deal with this.)
|
If I remember correctly since coreos-pool tag serves all of our streams coreos-koji-tagger will sign an fXX built rpm with the fXX key (this was the heuristic that made the most sense). What you're saying is that this causes issues because when we promote lockfiles from the bodhi stream where the rpms, even though they were built against f30, are signed with the f31 key and thus will have a different hash in the lockfile. Is that correct? |
|
/approve |
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: cgwalters, jlebon The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
bot, retest this please |
Right, that's one of the issues. I think this sums it up well: The Koji pkg is unsigned, the f31 version is signed with the f31 key, and the f30 and pool versions are signed with the f30 key. The main concern here is that in contexts where we're not locked, we could choose the package which isn't in the pool. Then we'd get a mismatch when moving the resulting lockfile to a context where we are locked. The main workflow that comes to mind that's like this is in lockfile promotion as you mentioned. |
|
The local dev workflow can also hit this when using |
One easy way to solve that problem is to only allow the pool repo as the input repo. As noted before this hinders development a bit.
Could we just promote the NVRAs instead of including the checksums? What do we gain or lose by doing that? |
The digest fields in lockfiles have been more trouble than they're worth so far. The major problem is signing: adding a signature to an RPM of course changes its digest. There are two main issues. The first is that when adding an override, there's no way to know the digest ahead of time since the package isn't signed yet. It becomes signed when the override is accepted and tagged into the pool. The second is that the same package NEVRA may be present in different repos and signed with different keys, resulting in different hashes, even if the payload is the same (for more details on this case, see coreos/rpm-ostree#1927). One could add repo origin information as part of the lockfile, but that does not correspond to what we actually do in FCOS, since packages initially come from Bodhi, and are then imported into the pool (and even then, there's no guarantee that the package isn't already there with a different signature). A possible solution is to instead use the CPIO payload digest, but that information is not part of the repo metadata, so it'd require some work for rpm-ostree to download the packages and select the right one. Not especially hard, but definitely non-trivial. In practice, we don't really need digests in the lockfile. For repos which don't use GPG checking, it could provide some security benefits. Though in our case, GPG verification is turned on for all the repos anyway. But more importantly, all our data comes from Koji, which forbids buildings two packages with the same NEVRA. We could fold this into rpm-ostree under a new flag, though I think its current behaviour makes sense overall. It just doesn't work well for us.
|
OK, follow-up to this in coreos/coreos-assembler#854. |
The digest fields in lockfiles have been more trouble than they're worth so far. The major problem is signing: adding a signature to an RPM of course changes its digest. There are two main issues. The first is that when adding an override, there's no way to know the digest ahead of time since the package isn't signed yet. It becomes signed when the override is accepted and tagged into the pool. The second is that the same package NEVRA may be present in different repos and signed with different keys, resulting in different hashes, even if the payload is the same (for more details on this case, see coreos/rpm-ostree#1927). One could add repo origin information as part of the lockfile, but that does not correspond to what we actually do in FCOS, since packages initially come from Bodhi, and are then imported into the pool (and even then, there's no guarantee that the package isn't already there with a different signature). A possible solution is to instead use the CPIO payload digest, but that information is not part of the repo metadata, so it'd require some work for rpm-ostree to download the packages and select the right one. Not especially hard, but definitely non-trivial. In practice, we don't really need digests in the lockfile. For repos which don't use GPG checking, it could provide some security benefits. Though in our case, GPG verification is turned on for all the repos anyway. But more importantly, all our data comes from Koji, which forbids buildings two packages with the same NEVRA. We could fold this into rpm-ostree under a new flag, though I think its current behaviour makes sense overall. It just doesn't work well for us.
Don't just filter down packages by NEVRA, but also filter out those that
don't match the checksum too. We were enforcing checksum matches already
before this, but only after depsolving and simply erroring out if they
didn't match.
However, because of how RPM signing is implemented in Fedora, it is
possible to have the same NEVRA in two different repos, each with two
different hashes. E.g. right now for example,
efivar-libswasn'trebuilt for f31, and so f31 is just shipping the f30 RPM, but signed
with the f31 key. And of course, we also had the f30 version in the
pool.
This patch allows us to transition over to the f31 version with
everything else by not getting thrown off by the f30 version already in
the pool. (Still need to investigate how the pool will deal with this.)