sota.conf.inc: remove SOURCE_DATE_EPOCH and REPRODUCIBLE_TIMESTAMP_ROOTFS

[shr-project]

• SOURCE_DATE_EPOCH set to 0 is now causing python3-setuptools-native build to fail with:

  creating build/bdist.linux-x86_64/wheel/setuptools-59.5.0.dist-info/WHEEL
  creating '/OE/lge/build/webosose/kirkstone/BUILD/work/x86_64-linux/python3-setuptools-native/59.5.0-r0/dist/tmpqgh1qmw6/setuptools-59.5.0-py3-none-any.whl' and adding 'build/bdist.linux-x86_64/wheel' to it
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/OE/lge/build/webosose/kirkstone/BUILD/work/x86_64-linux/python3-setuptools-native/59.5.0-r0/setuptools-59.5.0/setuptools/build_meta.py", line 230, in build_wheel
      return self._build_with_temp_dir(['bdist_wheel'], '.whl',
    File "/OE/lge/build/webosose/kirkstone/BUILD/work/x86_64-linux/python3-setuptools-native/59.5.0-r0/setuptools-59.5.0/setuptools/build_meta.py", line 215, in _build_with_temp_dir
      self.run_setup()
    File "/OE/lge/build/webosose/kirkstone/BUILD/work/x86_64-linux/python3-setuptools-native/59.5.0-r0/setuptools-59.5.0/setuptools/build_meta.py", line 158, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 87, in <module>
      dist = setuptools.setup(**setup_params)
    File "/OE/lge/build/webosose/kirkstone/BUILD/work/x86_64-linux/python3-setuptools-native/59.5.0-r0/setuptools-59.5.0/setuptools/__init__.py", line 153, in setup
      return distutils.core.setup(**attrs)
    File "/OE/lge/build/webosose/kirkstone/BUILD/work/x86_64-linux/python3-setuptools-native/59.5.0-r0/recipe-sysroot-native/usr/lib/python3.10/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/OE/lge/build/webosose/kirkstone/BUILD/work/x86_64-linux/python3-setuptools-native/59.5.0-r0/recipe-sysroot-native/usr/lib/python3.10/distutils/dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "/OE/lge/build/webosose/kirkstone/BUILD/work/x86_64-linux/python3-setuptools-native/59.5.0-r0/recipe-sysroot-native/usr/lib/python3.10/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/OE/lge/build/webosose/kirkstone/BUILD/work/x86_64-linux/python3-setuptools-native/59.5.0-r0/recipe-sysroot-native/usr/lib/python3.10/site-packages/wheel/bdist_wheel.py", line 361, in run
      wf.write_files(archive_root)
    File "/OE/lge/build/webosose/kirkstone/BUILD/work/x86_64-linux/python3-setuptools-native/59.5.0-r0/recipe-sysroot-native/usr/lib/python3.10/site-packages/wheel/wheelfile.py", line 136, in write_files
      self.write(path, arcname)
    File "/OE/lge/build/webosose/kirkstone/BUILD/work/x86_64-linux/python3-setuptools-native/59.5.0-r0/recipe-sysroot-native/usr/lib/python3.10/site-packages/wheel/wheelfile.py", line 147, in write
      zinfo = ZipInfo(arcname or filename, date_time=get_zipinfo_datetime(st.st_mtime))
    File "/OE/lge/build/webosose/kirkstone/BUILD/work/x86_64-linux/python3-setuptools-native/59.5.0-r0/recipe-sysroot-native/usr/lib/python3.10/zipfile.py", line 362, in __init__
      raise ValueError('ZIP does not support timestamps before 1980')
  ValueError: ZIP does not support timestamps before 1980

The st.st_mtime is correct timestamp, but get_zipinfo_datetime function in python3.10/site-packages/wheel/wheelfile.py
will prefer SOURCE_DATE_EPOCH when set:

  def get_zipinfo_datetime(timestamp=None):
      # Some applications need reproducible .whl files, but they can't do this without forcing
      # the timestamp of the individual ZipInfo objects. See issue #143.
      timestamp = int(os.environ.get('SOURCE_DATE_EPOCH', timestamp or time.time()))
      return time.gmtime(timestamp)[0:6]

Which leads to 1.1.1970 which isn't supported by ZIP.

  The default value will use get_source_date_epoch_value:
  # $SOURCE_DATE_EPOCH [2 operations]
  #   exported /OE/lge/build/webosose/kirkstone/oe-core/meta/conf/bitbake.conf:672
  #     [export] "1"
  #   set? /OE/lge/build/webosose/kirkstone/oe-core/meta/conf/bitbake.conf:672
  #     "${@get_source_date_epoch_value(d)}"
  # pre-expansion value:
  #   "${@get_source_date_epoch_value(d)}"
  export SOURCE_DATE_EPOCH="1638725317"

which in this case is:

  python3-setuptools-native/59.5.0-r0/source-date-epoch/__source_date_epoch.txt
  1638725317

  $ date --date='@1638725317'
  Sun Dec  5 18:28:37 CET 2021

I believe the reproducible builds now enabled by default in oe-core work
reasonably well and forcing this to 0 as added in:
3d3a0bf
af1694c
is no longer necessary.

Signed-off-by: Martin Jansa [email protected]

[pattivacek]

Have you tested this? I'd prefer if we could confirm that it still works as expected.

@agners what's your take on this, if you remember the context anymore?

[shr-project]

I've tested that it unblocks building python3-setuptools-native and the image with sota enabled was built successfully, but I don't have any setup to properly test incremental rootfs updates.

reproducible builds themselves (without meta-updater) are tested by yocto autobuilder

[agners]

From what I remember, the main concern was really this:

Python for instance writes the mtime of the source file into
bytecode files (pyc) at build time. When OSTree then clears
the mtime Python considers all bytecode files as stale.

As far as I remember, the problem is that OSTree really forces/sets the timestamp to 0. Building with timestamp forced to 0 makes sure that the timestamp in pyc files is 0 as well. This avoids Python recompiling Python files.

I believe the reproducible builds now enabled by default in oe-core work
reasonably well and forcing this to 0 as added in:
3d3a0bf
af1694c
is no longer necessary.

Is reproducible builds by default setting timestamps to zero? I assume not, so I'd say the above is still a problem.

Maybe patching OSTree is necessary (to use whatever timestamp OE reproducible build is using), or keeping timestamp at 0 and fix the python3-setuptools-native package somehow different.

[agners]

FWIW, background on why OSTree needs to mess with timestamps are outlined here: https://ostree.readthedocs.io/en/stable/manual/repo/

[vnd]

I've stumbled upon this issue simply by adding meta-arm to layers, along with meta-updater, and trying to build ARM trusted firmware. Would be grateful for a workaround that doesn't break anything else..

[shr-project]

Would be grateful for a workaround that doesn't break anything else..

I'm not aware of this PR breaking anything else, but I use only small portion of this layer

[vnd]

Would be grateful for a workaround that doesn't break anything else..

I'm not aware of this PR breaking anything else, but I use only small portion of this layer

Yeah, we've also applied it locally and it seems to work (thank you!), hope it will be merged eventually, or at least there will be a clear understanding of what it breaks.

[pattivacek]

we've also applied it locally and it seems to work

It definitely will result in valid output. Or did you mean that it wasn't recompiling the python modules as described above?

at least there will be a clear understanding of what it breaks.

I think it's reasonably clear, it's just that no one has dug into sorting out the last missing details. Or do you or @shr-project have new information here? I'm afraid I'm not well-versed here, but I remember the timestamp issues being troublesome some years ago.

I still feel a bit strange being the only person regularly reviews PRs here, because I only really have access to a dunfell system, and I don't hack on this much anymore. It would be lovely if someone who uses it more actively were able to share maintenance duties.

[ricardosalveti]

I believe the original pyc issue will happen back again when not setting up to 0, but 0 itself is indeed not perfect as not every software supports unix timestamps and build/runtime might indeed fail.

When not set to 0 oe will try to identify a static timestamp (e.g. last time source code changed) which works fine from the reproducible pov, but since this will be stuck into the pyc files, a mismatch will happen at runtime because due ostree mtime will be 0.

We can drop this change (as suggested by this pr) to avoid the build failures, but pyc optimizations will still be required to work the same way as before (as done by @agners ).

[pattivacek]

pyc optimizations will still be required to work the same way as before

As in, we need a new solution to that problem, or do we already have what we need?

[ricardosalveti]

As in, we need a new solution to that problem, or do we already have what we need?

We would need a new solution, and it might not be trivial because ostree uses unix timestamp and not every component supports it (from this error it looks like the issue is with ZIP itself - ZIP does not support timestamps before 1980). Since zip is commonly used by python when modules are created, it might be just easier to force mtime to be 0 (1970) when pyc files are created, but this would need further investigation in python.