Handling formulae which require reinstalling on major OS upgrades

[Bo98]

Background

A very small number of formulae have baked in references to the OS they were built on. For example:

• pkg-config bakes in a path to /usr/local/Homebrew/Library/Homebrew/os/mac/pkgconfig/10.15
• llvm/gcc bake in default sysroots to the CLT SDK matching the version it was built on.

This is normal and expected behaviour.

There is however a problem when the user upgrades their OS to a new major version. They will still have the formulae installed with 10.15 references and the user won't be getting the changes in the new bottle unless there is a version update/revision bump. As such, the software will not behave as desired (e.g. clang will continue to use 10.15 SDK despite running on 11.0). Usually the effects are not critical enough to cause a fire, but can cause some breakages like that demonstrated in #10079.

Paths forward

I'm opening this issue to discuss possible solutions. I've listed below some suggestions I think people may have:

Do revision bumps handle it?

No. Revision bumping occurs at a fixed time but users individually upgrading their OS doesn't happen at one fixed time. A timeline could easily be:

• Big Sur releases
• We revision bump pkg-config soon after.
• A few days later, Billy, on 10.15, runs brew upgrade which upgrades pkg-config.
• Billy then upgrades to 11.0 a few weeks later.
• Billy still has the 10.15 version of pkg-config because no update has been prompted after the point he upgraded to 11.0.

Can we avoid OS version references?

Of course, ideally we'd make such references dynamic and determined at runtime. This is however usually not possible without either patches or wrapper scripts, both which would be up to us to maintain. Such cases cannot be generalised and would be treated on a case-by-case basis so patches/scripts would vary between formulae.

For comparison, Apple's clang uses the wrapper script (well, actually a wrapper binary) approach in the form of xcrun and the /usr/bin shims.

The approach of patching is usually seen as a last resort so I don't anticipate this will be the favoured solution.

Prompting or automating a reinstall

A manual fix for affected formulae is brew reinstall <formula>. We could either heavily promote this or, even better, integrate it into brew so it happens as a part of brew upgrade or similar.

Since b338398, we now have the capability to see which OS version a formula was installed under. For selected formulae (either from a hardcoded list or, perhaps better, via DSL), we can see whether the runtime OS version mismatches the OS version in the tab and use that to suggest/run a reinstall.

Simply suggesting another command to run is a possibility, though recognising the difference between brew update and brew upgrade is hard enough for some. Having to remember another command is awkward.

An optimal solution is one that doesn't require extra steps from the user. Ideally, a user should just be able to do brew upgrade and everything is done for them. Perhaps even have brew outdated print out formulae that need a reinstall. I do however realise that there is a distinct difference between reinstalling the same version (albeit a different build) and upgrading and that this may blur the lines a little.

If that blurring is a problem, then a potential possiblity is to incorporate the OS version into the version itself (e.g. Debian style ~11.0 suffix) - if that makes more sense internally. That means that some existing upgrade logic could apply.

[Bo98]

(I continue to be inactive for a few more days at least, but will try keep up with discussion here - I just may be a little slow to respond. Ping me on Slack if you need to grab my attention more quickly - I have notifications on.)

[carlocab]

Ideally, a user should just be able to do brew upgrade and everything is done for them. Perhaps even have brew outdated print out formulae that need a reinstall.

This would be ideal, I think. Do we have a way of systematically identifying formulae that will need to be reinstalled upon an OS upgrade?

[Bo98]

We would need to either make a list somewhere or have a DSL to mark such formulae. It should only be a handful of formulae.

From such list we can very easily compare the install receipt/tab with the runtime macOS versison to see what hasn't been reinstalled yet.

[carlocab]

It should only be a handful of formulae.

Yes, I was asking how we would identify that handful of formulae to put in a list or mark in some DSL.

[Bo98]

I suppose scanning files for MacOS.version references is a starting point. But a lot of cases might just be build information stuff baked into the binary that doesn't matter.

[MikeMcQuaid]

We would need to either make a list somewhere or have a DSL to mark such formulae. It should only be a handful of formulae.

I think we could maintain a list in Homebrew/brew for now.

I like the idea of having brew upgrade automatically perform any reinstalls (much like how we verify dependencies are up-to-date and dependents don't have broken linkage).

[Bo98]

Honestly the only "big ones" I had on mind were:

• pkg-config
• llvm (all versions)
• gcc (all versions)
• python (all versions) - see python3-config --cflags
• vtk
• root

Any reported breakages from the 11.0 SDK removal should reveal any others too (even though that's a separate problem).

[carlocab]

abseil seems to have the SDK path baked in as well. Homebrew/homebrew-core#67474

[carlocab]

Ran into a similar issue here: Homebrew/homebrew-core#67615

My working guess at the moment is that any formulae with BUILD_SHARED_LIBS=ON is a candidate for related problems. These are the formulae that do this:

Formulae

abseil
ampl-mp
bullet
caffe
cctz
ceres-solver
cminpack
cpr
crc32c
dcmtk
dlib
double-conversion
f3d
fizz
fmt
folly
freerdp
gdcm
gflags
glfw
glog
grpc
h3
howard-hinnant-date
itk
lapack
leveldb
libcbor
libcue
libebml
libevhtp
libgit2
libilbc
libmatroska
liboqs
libsignal-protocol-c
matplotplusplus
neovim
netcdf
nng
opencv
opencv@3
osquery
pcl
poppler
re2
s2geometry
s2n
scalapack
seal
snappy
stormlib
sundials
taglib
teem
tmx
vera++
vtk
[email protected]
wangle
xerces-c
yaml-cpp
zeek

[MikeMcQuaid]

My thoughts on the best fit for this would be to search for the SDK reference (see each_unique_file_matching in keg_relocate.rb) and fail a brew audit --installed if it either isn't part of some hardcoded list we have in Homebrew/brew or has some DSL set.

This list/DSL would then be used for the aforementioned reinstall logic.

[carlocab]

Two Four more related issues:
Homebrew/homebrew-core#68038
Homebrew/homebrew-core#68049
Homebrew/homebrew-core#67306 (vtk has the SDK path baked in)
Homebrew/homebrew-core#68877 (libccd)

I'm going to consolidate my various comments pointing to related issues into a single one at some point. Apologies for the extra notifications; this should be the last one.

[lstomberg]

The git-svn script as part of the git formulae uses the OS perl, which changed from v5.18 for almost all of 10.x to v5.28 in macOS Big Sur 11.0-11.2 and just today changed to v5.30 in 11.3. This has caused a ton of headaches, both for my company and around the Internet forums on how best to handle this.

subversion also installs perl bindings based on the perl version on the system. The macOS 10.15 bottle includes 5.18.2 bindings, installed in the directory /usr/local/opt/subversion/lib/perl5/site_perl/5.18.2, whereas macOS 11.0 instead has 5.28.2 bindings. The formulae prefers /usr/local/bin/perl over /usr/bin/perl, so building subversion locally while brew's perl v5.32.1 is installed will output 5.32.1 bindings.

[Bo98]

today changed to v5.30 in 11.3

Apple did a major update to Perl in a minor macOS update? That's very unusual and will indeed break things.

We only have one bottle for all Big Sur versions so will need to try maintain compatibility for all of 11.x.

[lstomberg]

@Bo98 I saw the downloaded bottle simply said "BigSur" without a version number, so I was guessing there was probably only a single bottle for all of 11.x. The perl change and the versioning for bottles sounds like it will cause very big problems. If you had to hazard a guess, do you think these issues will be able to be resolved in the next week or could it take multiple weeks? I'm just trying to plan internally at my company what to do to handle this.

[Bo98]

I'll be discussing the situation with other maintainers in a couple hours and will report back what the plan is.

[Bo98]

Ok so 5.30 is present on earlier Big Sur versions. What has changed is the default version of Perl.

We do plan to do some Perl shebang rewriting as a part of #11227 so we could rewrite everything on Big Sur to use /usr/bin/perl5.30 rather than rely on the default /usr/bin/perl. We can then rebuild the Big Sur bottle to use Perl 5.30. Though relocation doesn't apply to building from source so there's more to think about here.

[Bo98]

Since the Perl issue is getting out of scope for what this issue is about, I've opened #11275 to track this properly.

[MikeMcQuaid]

@Bo98 What was/is the latest here?

[MikeMcQuaid]

Closing this out; it's either done or wasn't a big problem for the Monterey release.

[Bo98]

Wasn't a problem for Monterey because hardly anything changed in the OS and its SDK. I will address this eventually but it's not a priority just now.

[MikeMcQuaid]

@Bo98 👍🏻