Set SOURCE_DATE_EPOCH #58
Conversation
|
/cc @bmwiedemann |
suse-buildsystem.sh
Outdated
| @@ -7,3 +7,31 @@ export QT_HASH_SEED=0 | |||
| export PERL_HASH_SEED=42 | |||
| export PYTHONHASHSEED=0 | |||
| export FORCE_SOURCE_DATE=1 # for texlive to use SOURCE_DATE_EPOCH | |||
|
|
|||
| ( if test -f /.buildenv && test "Y" = "$(rpm --eval '%disable_obs_set_source_date_epoch')"; then | |||
There was a problem hiding this comment.
The code says to do this when %disable_obs_set_source_date_epoch = Y - that seems wrong.
I think, we should keep this "adding of release" off by default as it will break build-compare and build-tree-pruning for a large number of packages that already use SOURCE_DATE_EPOCH as timestamp.
So how about if test -f /.buildenv && test "Y" = "$(rpm --eval '%obs_set_source_date_epoch')"
There was a problem hiding this comment.
Thx, I need to fix the test.
Where do you expect default off to help? Either we publish and thus need this enabled or we do not do much build-compare based dependency-tree rebuild pruning because the number of packages are small because it is not a full distribution.
In either case the real solution is to fix upstream to not put dates in the output. Depending on the case if something is really needed there is usually something that is more meaningful. E.g. you could put the major version of the software instead. And for workarounds in spec files set SDE to 1 for a subprocess where you ensured it is not used for a file mtime. Either of these will result in build-compare still working.
There was a problem hiding this comment.
Both BUILD_CHANGELOG_TIMESTAMP and git commit mtime are also still available inside the spec file.
There was a problem hiding this comment.
For ease of use I leave it as BROKEN_SOURCE_DATE_EPOCH in the environment.
JanZerebecki
force-pushed
the
set_source_date_epoch
branch
4 times, most recently
from
e2e6460 to
e5294ae
Compare
|
@dirkmueller are you fine with merging this? |
|
The required obs-build patch is deployed on build.opensuse.org and it works. |
| if test -z SOURCE_DATE_EPOCH; then | ||
| unset SOURCE_DATE_EPOCH | ||
| else | ||
| export SOURCE_DATE_EPOCH |
There was a problem hiding this comment.
this might be a multiline string then with the subshell approach and warnings above. I am not sure this is useful behavior.
why are you doing this so complicated rather than simply setting SOURCE_DATE_EPOCH'?
There was a problem hiding this comment.
Using a subshell, because otherwise even non-exported variables would remain in the environment. So the only alternative I can come up with is to manually parse the .buildenv file and to unset every created intermediary variable. The subshell approach seemed easier to me.
None of the warnings are captured as they go to stderr, thus they will be seen in the buildlog.
JanZerebecki
force-pushed
the
set_source_date_epoch
branch
from
e5294ae to
a76ce02
Compare
replacing rpm its %source_date_epoch_from_changelog . This instead also increments the date by the build counter, which OBS puts into the release field of rpm. When available it uses the mtime that OBS its scmsync obtains from the git HEAD commit date instead of from the changelog. All this can be disabled by setting the rpm macro %disable_obs_set_source_date_epoch to Y . This uses new variables in /.buildenv that were implemented in openSUSE/obs-build@5d1da85 . See https://reproducible-builds.org/docs/source-date-epoch/ for general introduction. Increasing the date by the build counter is necessary, so that that even for rebuilds with unchanged source but updated build depends the date is always increased, such that no build with a different input/output has the same date. Not increasing the date breaks rsync without --checksum or anything else that relies on modification time stamps of files to detect changes. That the build counter is the number after the first dot of the release is a OBS convention. This breaks if a Release is set manually in the spec file. When %clamp_mtime_to_source_date_epoch and %source_date_epoch_from_changelog are enabled and a rebuild is done with different build dependencies a file may have different content but with the same mtime. When two releases of this rpm are extracted after each other at the same location and the extracted files are synced with rsync without --checksum, then the resulting copy will contain a mix of files from the old and new build. This can happen on installed systems due to updates when it is then backed up with rsync. While it is not safe generally safe to rely on mtimes and thus nor to backup systems without --checksum, we should not unnecessarily break things. We came across this problem because this happens for rpm repos from OpenSUSE that have extracted files used for installing from a repo, which get broken when mirroring with rsync, see https://bugzilla.suse.com/show_bug.cgi?id=1148824 .
JanZerebecki
force-pushed
the
set_source_date_epoch
branch
from
a76ce02 to
e56b51b
Compare
|
this code has a bug verbose version: |
|
Dirk fixed it in #59 |
|
This broke build-compare horribly as SOURCE_DATE_EPOCH now changes on spurious rebuilds: https://bugzilla.opensuse.org/show_bug.cgi?id=1216691 |
|
And is broken WRT time zones: |
|
Thanks for the reports. the build-compare clash is something I feared about and don't see easy to fix. I think the whole approach needs to be reconsidered. maybe we need to opt out for certain packages that need to have a meaningful rebuild. I have reverted the patch and submitted that revert to factory. @JanZerebecki what's your suggestion to approach those issues? |
|
tbh this should probably be in the build package and not be side injected by post-build-checks anyway |
|
I'll revisit this next year, won't have time before.
If there is a fix for something that was included from a build dependency then that would lead to those packages not receiving that fix. This also still has the problem that we need to touch every package that is affected. Considering all alternatives that were suggested, I'm afraid the only choice we have is how easy to spot the brokenness in affected packages are. Seems like packages changing on each increment is the least bad breakage and easiest to spot. Maybe I should ask r-b people from other distributions about this. Not sure if anyone else is doing automatic rebuilds. Wonder how nixos and guix do this.
The opposite was decided here: openSUSE/obs-build#955 (comment) |
|
I suspect there are few packages that set file mtimes in build scripts, so we can put a check in place that fails if the resulting rpm contains any file whose mtime is not SOURCE_DATE_EPOCH, without it triggering as often as SDE embedding preventing build-compare. Then we can probably overwrite SDE with the broken one inside the spec file so rpm still follows the outer one. While that currently violates the spec and it can theoretically cause a problem if SDE is used as a value in a protocol, I do not know of any case of that, maybe the chance of that is small enough. |
|
meant doing it for all packages, without individually checking each |
when computing the time of a changes entry in changelog2spec. This avoids getting a date in the future which would fail when using it for SOURCE_DATE_EPOCH for reproducible builds. Such failures happened when we tried using it with openSUSE/post-build-checks#58 . Example failure from build log: setting SOURCE_DATE_EPOCH to 1698749844 ERROR SOURCE_DATE_EPOCH is in the future, clamping mtime if used might fail in hard to notice way, returning error "Mon Oct 30 12:36:34 CET 2023" in .changes results in SOURCE_DATE_EPOCH=1698669395 but that's Mon 30. Oct 12:36:35 UTC 2023 Fixes: https://bugzilla.opensuse.org/show_bug.cgi?id=1216738
replacing rpm its %source_date_epoch_from_changelog . This instead also increments the date by the build counter, which OBS puts into the release field of rpm. When available it uses the mtime that OBS its scmsync obtains from the git HEAD commit date instead of from the changelog. All this can be disabled by setting the rpm macro %disable_obs_set_source_date_epoch to Y . This uses new variables in /.buildenv that were implemented in openSUSE/obs-build@5d1da85 . See https://reproducible-builds.org/docs/source-date-epoch/#interaction-of-source_date_epoch-with-automatic-rebuilds for more general discussion. Increasing the date by the build counter is necessary, so that that even for rebuilds with unchanged source but updated build depends the date is always increased, such that no build with a different input/output has the same date. Not increasing the date breaks rsync without --checksum or anything else that relies on modification time stamps of files to detect changes. That the build counter is the number after the first dot of the release is a OBS convention. This breaks if a Release is set manually in the spec file. When %clamp_mtime_to_source_date_epoch and %source_date_epoch_from_changelog are enabled and a rebuild is done with different build dependencies a file may have different content but with the same mtime. When two releases of this rpm are extracted after each other at the same location and the extracted files are synced with rsync without --checksum, then the resulting copy will contain a mix of files from the old and new build. This can happen on installed systems due to updates when it is then backed up with rsync. While it is not safe generally safe to rely on mtimes and thus nor to backup systems without --checksum, we should not unnecessarily break things. We came across this problem because this happens for rpm repos from OpenSUSE that have extracted files used for installing from a repo, which get broken when mirroring with rsync, see https://bugzilla.suse.com/show_bug.cgi?id=1148824 . This needs openSUSE/obs-build#977 to be deployed first. This works together with rpm-software-management/rpm#2880 . For the full effect its settings need to be enabled. First try openSUSE#58 was reverted with bfa988b .
replacing rpm its %source_date_epoch_from_changelog . This instead also increments the date by the build counter obtained from OBS. When available it uses the mtime that OBS its scmsync obtains from the git HEAD commit date instead of from the changelog. All this can be disabled by setting the rpm macro %disable_obs_set_source_date_epoch to Y . This uses new variables in /.buildenv that were implemented in openSUSE/obs-build@5d1da85 . See https://reproducible-builds.org/docs/source-date-epoch/#interaction-of-source_date_epoch-with-automatic-rebuilds for more general discussion. Increasing the date by the build counter is necessary, so that that even for rebuilds with unchanged source but updated build depends the date is always increased, such that no build with a different input/output has the same date. Not increasing the date breaks rsync without --checksum or anything else that relies on modification time stamps of files to detect changes. When %clamp_mtime_to_source_date_epoch and %source_date_epoch_from_changelog are enabled and a rebuild is done with different build dependencies a file may have different content but with the same mtime. When two releases of this rpm are extracted after each other at the same location and the extracted files are synced with rsync without --checksum, then the resulting copy will contain a mix of files from the old and new build. This can happen on installed systems due to updates when it is then backed up with rsync. While it is not safe generally safe to rely on mtimes and thus nor to backup systems without --checksum, we should not unnecessarily break things. We came across this problem because this happens for rpm repos from OpenSUSE that have extracted files used for installing from a repo, which get broken when mirroring with rsync, see https://bugzilla.suse.com/show_bug.cgi?id=1148824 . This needs openSUSE/obs-build#977 to be deployed first. This works together with rpm-software-management/rpm#2880 . For the full effect its settings need to be enabled. First try openSUSE#58 was reverted with bfa988b .
replacing rpm its %source_date_epoch_from_changelog . This instead also increments the date by the build counter obtained from OBS. When available it uses the mtime that OBS its scmsync obtains from the git HEAD commit date instead of from the changelog. It exports the old date in SOURCE_DATE_EPOCH and the new incremented date in SOURCE_DATE_EPOCH_MTIME. rpm will use SOURCE_DATE_EPOCH_MTIME for for file mtimes and the old date for everything else. All this can be disabled by setting the rpm macro %disable_obs_set_source_date_epoch to Y . This uses new variables in /.buildenv that were implemented in openSUSE/obs-build@5d1da85 . See https://reproducible-builds.org/docs/source-date-epoch/#interaction-of-source_date_epoch-with-automatic-rebuilds for more general discussion. Increasing the date by the build counter is necessary, so that that even for rebuilds with unchanged source but updated build depends the date is always increased, such that no build with a different input/output has the same date. Not increasing the date breaks rsync without --checksum or anything else that relies on modification time stamps of files to detect changes. When %clamp_mtime_to_source_date_epoch and %source_date_epoch_from_changelog are enabled and a rebuild is done with different build dependencies a file may have different content but with the same mtime. When two releases of this rpm are extracted after each other at the same location and the extracted files are synced with rsync without --checksum, then the resulting copy will contain a mix of files from the old and new build. This can happen on installed systems due to updates when it is then backed up with rsync. While it is not safe generally safe to rely on mtimes and thus nor to backup systems without --checksum, we should not unnecessarily break things. We came across this problem because this happens for rpm repos from OpenSUSE that have extracted files used for installing from a repo, which get broken when mirroring with rsync, see https://bugzilla.suse.com/show_bug.cgi?id=1148824 . This needs openSUSE/obs-build#977 to be deployed first. This works together with rpm-software-management/rpm#2880 . For the full effect its settings need to be enabled. First try openSUSE#58 was reverted with bfa988b .
replacing rpm its %source_date_epoch_from_changelog . This instead also increments the date by the build counter obtained from OBS. When available it uses the mtime that OBS its scmsync obtains from the git HEAD commit date instead of from the changelog. It exports the old date in SOURCE_DATE_EPOCH and the new incremented date in SOURCE_DATE_EPOCH_MTIME. rpm will use SOURCE_DATE_EPOCH_MTIME for for file mtimes and the old date for everything else. All this can be disabled by setting the rpm macro %disable_obs_set_source_date_epoch to Y . This uses new variables in /.buildenv that were implemented in openSUSE/obs-build@5d1da85 . See https://reproducible-builds.org/docs/source-date-epoch/#interaction-of-source_date_epoch-with-automatic-rebuilds for more general discussion. Increasing the date by the build counter is necessary, so that that even for rebuilds with unchanged source but updated build depends the date is always increased, such that no build with a different input/output has the same date. Not increasing the date breaks rsync without --checksum or anything else that relies on modification time stamps of files to detect changes. When %clamp_mtime_to_source_date_epoch and %source_date_epoch_from_changelog are enabled and a rebuild is done with different build dependencies a file may have different content but with the same mtime. When two releases of this rpm are extracted after each other at the same location and the extracted files are synced with rsync without --checksum, then the resulting copy will contain a mix of files from the old and new build. This can happen on installed systems due to updates when it is then backed up with rsync. While it is not safe generally safe to rely on mtimes and thus nor to backup systems without --checksum, we should not unnecessarily break things. We came across this problem because this happens for rpm repos from OpenSUSE that have extracted files used for installing from a repo, which get broken when mirroring with rsync, see https://bugzilla.suse.com/show_bug.cgi?id=1148824 . This needs openSUSE/obs-build#977 to be deployed first. This works together with rpm-software-management/rpm#2880 . For the full effect its settings need to be enabled. First try openSUSE#58 was reverted with bfa988b .
replacing rpm its %source_date_epoch_from_changelog . This instead also increments the date by the build counter, which OBS puts into the release field of rpm. When available it uses the mtime that OBS its scmsync obtains from the git HEAD commit date instead of from the changelog.
All this can be disabled by setting the rpm macro
%disable_obs_set_source_date_epoch to Y .
This uses new variables in /.buildenv that were implemented in openSUSE/obs-build@5d1da85 .
See https://reproducible-builds.org/docs/source-date-epoch/ for general introduction.
Increasing the date by the build counter is necessary, so that that even for rebuilds with unchanged source but updated build depends the date is always increased, such that no build with a different input/output has the same date.
Not increasing the date breaks rsync without --checksum or anything else that relies on modification time stamps of files to detect changes.
That the build counter is the number after the first dot of the release is a OBS convention. This breaks if a Release is set manually in the spec file.
When %clamp_mtime_to_source_date_epoch and
%source_date_epoch_from_changelog are enabled and a rebuild is done with different build dependencies a file may have different content but with the same mtime. When two releases of this rpm are extracted after each other at the same location and the extracted files are synced with rsync without --checksum, then the resulting copy will contain a mix of files from the old and new build. This can happen on installed systems due to updates when it is then backed up with rsync. While it is not safe generally safe to rely on mtimes and thus nor to backup systems without --checksum, we should not unnecessarily break things. We came across this problem because this happens for rpm repos from OpenSUSE that have extracted files used for installing from a repo, which get broken when mirroring with rsync, see
https://bugzilla.suse.com/show_bug.cgi?id=1148824 .