[CC7] cmsTriton fails to start container: /lib64/libc.so.6: version `GLIBC_2.28'

[iarspider]

In CMSSW_14_1_X_2024-07-14-0000 for slc7_amd64_gcc12, unit tests and relvals that use Triton are failing:

/cvmfs/oasis.opensciencegrid.org/mis/apptainer/1.3.3/x86_64/bin/apptainer: /lib64/libc.so.6: version `GLIBC_2.28' not found (required by /cvmfs/oasis.opensciencegrid.org/mis/apptainer/1.3.3/x86_64/bin/apptainer)
/cvmfs/oasis.opensciencegrid.org/mis/apptainer/1.3.3/x86_64/bin/apptainer: /lib64/libc.so.6: version `GLIBC_2.28' not found (required by /cvmfs/oasis.opensciencegrid.org/mis/apptainer/1.3.3/x86_64/bin/apptainer)

(full log: link ).

Seems to be caused by update of /cvmfs/oasis.opensciencegrid.org/mis/apptainer/current symlink to a version that doesn't support CC7.

@kpedro88 FYI

[iarspider]

assign heterogeneous

[cmsbuild]

New categories assigned: heterogeneous

@fwyzard,@makortel you have been requested to review this Pull request/Issue and eventually sign? Thanks

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @iarspider.

@Dr15Jones, @antoniovilela, @makortel, @rappoccio, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[makortel]

I'm not too worried on these tests starting to fail on slc7 (that is EOL and we should find a way to be able to drop those builds) as such, but such tests starting to fail independently of any CMS' action sounds worrisome.

[kpedro88]

I've submitted a ticket to OSG to ask if they can retain support for sl7, but I'm not sure what they'll say.

The use of OSG apptainer was to avoid being frequently stuck with bugs that would break apptainer-in-apptainer until CMS containers were updated and distributed, which is not currently a fast process across all sites, cvmfs repos, etc. I agree this could be improved in several ways:

1. CMS could distribute its own up-to-date apptainer in a more responsive way
2. avoiding apptainer-in-apptainer altogether (on the SONIC roadmap, but not achieved yet)

[kpedro88]

Report from OSG ticket: the apptainer developers do not promise support for sl7 in new versions. OSG recommends pinning to 1.2.5, which is the last version guaranteed to work for all sl7 distributions (1.3.0 potentially broke some things, though up to 1.3.2 had worked in my personal tests). Therefore, I will pin to that version when sl7 is detected.