2 glibc issues in the past 24 hours

[dannf]

We have experienced 2 issues with GLIBC in the past 24 hours. Both have been resolved and the archive is back to normal. Here is what happened:

Issue 1: GLIBC 2.40 compile option hardening caused issues starting certain binaries
Yesterday we rebuilt our existing GLIBC 2.40 package with OpenSSF-recommended hardening options enabled. We had enabled these options distro-wide last year, but explicitly disabled those options when building GLIBC because it failed to build on ARM. The build failure on ARM was due to some missing support in our GCC builds. We since addressed the missing support in GCC, so yesterday we attempted to enable the OpenSSF hardening in GLIBC. Unfortunately those hardening flags ended up causing problems with the dynamic loader that caused certain binaries on x86 to fail to start. While we believe we understand the root cause and necessary fix, we chose to address this by temporarily re-disabling the OpenSSF hardening in GLIBC for now while we perform additional testing and add additional automated tests to avoid this type of failure in the future.

Time	Event
2025-01-29 01:23 UTC	OpenSSF-hardened GLIBC 2.40 (2.40-r7) enters archive
2025-01-29 16:23 UTC	Issues reported to development team
2025-01-29 17:11 UTC	Development team identifies GLIBC change as root cause
2025-01-29 17:23 UTC	OpenSSF-hardening re-disabled in GLIBC (2.40-r8)

Issue 2: GLIBC 2.41 update ordering temporarily broke wolfi builds
After resolving the above issue today, we moved ahead with a planned upgrade to GLIBC 2.41. We had been preparing for this expected update for some time, including the testing of full archive rebuilds. However, due to a different ordering of operations, deploying this update into wolfi ended up breaking our build machinery. We decided the best course of action was to withdraw this update. We will delay the 2.41 update for now while we adapt our plan to account for the ordering issues.

Time	Event
2025-01-29 18:27 UTC	GLIBC 2.41 introduced into archive (2.41-r1)
2025-01-29 18:40 UTC	GLIBC 2.41 roll-out halted due to build-ordering issues
2025-01-28 20:32 UTC	GLIBC 2.41 roll-out reverted

[ibuildthecloud]

About an hour ago these two files were out of sync: /usr/include/pthread.h /usr/lib/gcc/x86_64-pc-linux-gnu/14/include-fixed/pthread.h

And now they are in sync and my build works again. So whatever happened thanks for fixing it fast.

[xnox]

@ibuildthecloud yes, thank you for that tweet as it is the underlying cause that will help us to fix issue 2, because we were perplexed by gcc bootstrap no longer able to complete, and yes the pthread.h out of sync is the culprit. Once we resolve that we will attempt to land glibc 2.41 again.

[kaniini]

The include-fixed files should probably just be deleted. GCC can compile the normal glibc headers.

[xnox]

The include-fixed files should probably just be deleted. GCC can compile the normal glibc headers.

Yeah, but what is confusing us, is why we didn't have them before, didn't have them in the 2.41 pre release snapshot rebuild, and it exploded upon landing it.

Maybe it is too boring to find out and we should just rm -rf them.