-
Notifications
You must be signed in to change notification settings - Fork 169
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Builds are not reproducible #3332
Comments
Bit-level reproducibility is currently not a goal of coreos-assembler. Given the work that went into rpm-ostree to make composes reproducible, it probably wouldn't be too hard to at least make But cosa is used to also build many other artifacts (disk images and container images). Trying to make those fully reproducible would be a very large endeavour. |
The issue I'm reporting here is that the ostree commits aren't reproducible either. |
No, it hasn't really been a focus, but it'd definitely be nice to support. That would also allow reproducible builds of the container image. I know you're using bdb, but for general information, note that for sqlite (which is what FCOS and el9 use), actually achieving this is also blocked on rpm-software-management/rpm#2219. |
I won't speak for the other maintainers but I'd accept patches to enable this at least for the ostree compose (assuming they're not invasive, which I don't think they would be). |
For posterity, I traced this to 2 issues. The first is that some Python libraries are byte-compiled in the root file system that's part of ostree, but only if they're used as part of an rpm installation hook for one of the included RPMs, and Python byte compilation has never been reproducible. The only current fix for this is to manually wipe all python byte-compilation caches before the final ostree compose using a custom postprocess:
- |
echo "Removing all module __pycache__ folders under /usr"
find /usr -type d -name '__pycache__' -exec rm -r '{}' + The second issue is that the boot image is included in the ostree commits generated, and the boot image is a binary blob. That binary blob is not reproducible (I haven't figured out the exact reasons why yet), but it's causing the resulting ostree commit that's created containing it to be different. |
Bug Report
This might be an
ostree
bug?Builds using the coreos-assembler container with
-e SOURCE_DATE_EPOCH=${fixed_epoch}
, a fixed config commit, and run withbuild --version=${fixed_buildid}
on a pristine VM system are completing but producing differentostree-content-checksum
values (permeta.json
) on every build repetition ofcosa clean && cosa build --version=${fixed_buildid}
.Environment
What operating system is being used to run coreos-assembler?
Fedora 37 (in a fresh VM)
What operating system is being assembled?
Customized minimal FCOS variant (fixed config commit that builds successfully)
Is coreos-assembler running in Podman or Docker?
podman 4.3.1
If Podman, is coreos-assembler running privileged or unprivileged?
privileged
Expected Behavior
Running
cosa clean
thencosa build --version=${fixed_version}
on a clean config repo withSOURCE_DATE_EPOCH
set to a fixed value should produce the sameostree-content-checksum
as the prior identical build when therpm-ostree-inputhash
is the same.Actual Behavior
Running
cosa clean
thencosa build --version=${fixed_version}
on a clean config repo withSOURCE_DATE_EPOCH
set to a fixed value produces a uniqueostree-content-checksum
on every build even when therpm-ostree-inputhash
is the same on all prior builds.Reproduction Steps
Using cosa alias:
cosa init --branch=my-fixed-branch git@private-gitlab/my-private-coreos-config.git
cosa fetch --strict
cosa build --version=59ff54f
cp builds/latest/x86_64/meta.json ~/
cosa clean
cosa fetch --strict # doesn't end up doing anything
cosa build --version=59ff54f
jq -f ~/meta.json '.ostree-content-checksum' > ~/build1.hash
jq -f builds/latest/x86_64/meta.json '.ostree-content-checksum' > ~/build2.hash
diff ~/build1.hash ~/build2.hash
Other Information
The CoreOS Config contains the following to make the RPM DB generation deterministic:
When comparing the entire meta.json files, the following are identical between the two builds
ref
ostree-n-metadata-total
ostree-n-metadata-written
ostree-n-content-total
ostree-n-content-written
ostree-n-cache-hits
ostree-content-bytes-written
ostree-version
(also matchesbuildid
)ostree-timestamp
rpm-ostree-inputhash
buildid
(also matchesostree-version
)coreos-assembler.image-genver
name
summary
coreos-assembler.image-config-checksum
"coreos-assembler.code-source": "container"
coreos-assembler.container-config-git
"coreos-assembler.delayed-meta-merge": false
coreos-assembler.container-image-git
"coreos-assembler.config-gitrev": "59ff54f0574445ef2912d7ecf1ccda71f0eb3efb"
"coreos-assembler.config-dirty": "false",
"coreos-assembler.basearch": "x86_64"
Trying lots of iterations, it appears this gets worse the larger the image size involved. Our configs that have a handful of the larger rpms commented out to reduce the resulting RPM total size from 6.8 GB to 500 MB have reproducible builds most of the time, though occasionally it will suddenly start producing different results, but the larger images never produce the same results.
The text was updated successfully, but these errors were encountered: