another apptainer workaround

[kpedro88]

PR description:

Another apptainer issue has appeared when starting the local Triton fallback server for el7 CMSSW (when running on el9 host OS). The symptom is:

INFO:    Disabling cgroups because systemd is unavailable
ERROR:   container cleanup failed: no instance found with name triton_server_instance_CPU_ee4479ae-3b48-43cb-8cb6-2524794d089e
FATAL:   container creation failed: while applying cgroups config: while creating cgroup manager: systemd not running on this host, cannot use systemd cgroups manager

This is not the exact same message as in apptainer/apptainer#2164, but the same workaround applies.

PR validation:

Unit tests now pass.

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

Will be backported to 14_0_X to fix the same issue.

[cmsbuild]

cms-bot internal usage

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-45652/41183

[cmsbuild]

A new Pull Request was created by @kpedro88 for master.

It involves the following packages:

• HeterogeneousCore/SonicTriton (heterogeneous)

@cmsbuild, @fwyzard, @makortel can you please review it and eventually sign? Thanks.
@makortel, @missirol, @riga, @rovere this is something you requested to watch as well.
@antoniovilela, @mandrenguyen, @rappoccio, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

[kpedro88]

please test

[makortel]

@cmsbuild, please test for slc7_amd64_gcc12

[cmsbuild]

+1

Size: This PR adds an extra 24KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-42e435/40805/summary.html
COMMIT: a614beb
CMSSW: CMSSW_14_1_X_2024-08-06-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/45652/40805/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially removed 3 lines from the logs
• Reco comparison results: 4 differences found in the comparisons
• DQMHistoTests: Total files compared: 45
• DQMHistoTests: Total histograms compared: 3423977
• DQMHistoTests: Total failures: 0
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3423957
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 44 files compared)
• Checked 196 log files, 165 edm output root files, 45 DQM output files
• TriggerResults: no differences found

[cmsbuild]

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-42e435/40807/summary.html
COMMIT: a614beb
CMSSW: CMSSW_14_1_X_2024-08-05-2300/slc7_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/45652/40807/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found 1 errors in the following unit tests:

---> test test_MC_23_setup had ERRORS

Comparison Summary

Summary:

• You potentially added 278 lines to the logs
• ROOTFileChecks: Some differences in event products or their sizes found
• Reco comparison results: 65359 differences found in the comparisons
• DQMHistoTests: Total files compared: 45
• DQMHistoTests: Total histograms compared: 3423977
• DQMHistoTests: Total failures: 222280
• DQMHistoTests: Total nulls: 404
• DQMHistoTests: Total successes: 3201273
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 2.093 KiB( 44 files compared)
• DQMHistoSizes: changed ( 13034.0 ): 1.523 KiB SiStrip/MechanicalView
• DQMHistoSizes: changed ( 140.043 ): -0.004 KiB JetMET/SUSYDQM
• DQMHistoSizes: changed ( 141.042 ): 0.008 KiB JetMET/SUSYDQM
• DQMHistoSizes: changed ( 141.044 ): -0.035 KiB JetMET/SUSYDQM
• DQMHistoSizes: changed ( 250202.181 ): -0.059 KiB SiStrip/MechanicalView
• DQMHistoSizes: changed ( 25202.0 ): 0.660 KiB SiStrip/MechanicalView
• Checked 196 log files, 165 edm output root files, 45 DQM output files
• TriggerResults: found differences in 15 / 43 workflows

[makortel]

slc7 unit test failure should get fixed with #45614

[makortel]

+heterogeneous

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @sextonkennedy, @rappoccio, @mandrenguyen, @antoniovilela (and backports should be raised in the release meeting by the corresponding L2)

[mandrenguyen]

+1

[mandrenguyen]

Is this 13_2_X IB failure related?
https://cmssdt.cern.ch/SDT/cgi-bin/logreader/slc7_amd64_gcc11/CMSSW_13_2_X_2024-08-14-2300/pyRelValMatrixLogs/run/10804.31_SingleGammaPt10+2018_photonDRN/step3_SingleGammaPt10+2018_photonDRN.log#/

[kpedro88]

@mandrenguyen indeed, thanks for noticing this. I have submitted another backport at #45710.