Remove tensorflow::setLogging() as thread-unsafe

[makortel]

PR description:

The setLogging() calls setenv(), which is not required to be thread safe, and specifically in glibc leads to a race condition with any concurrent getenv() calls. For more information see #46002 (comment). There is circumstantial evidence these specific setenv() calls could be causing the rare crash reported in #44659.

This PR should probably be accompanied with a PR to cmsdist setting TF_CPP_MIN_LOG_LEVEL=3 in the Tensorflow toolfile.

Resolves cms-sw/framework-team#1030

PR validation:

Code compiles, and tracing (with gdb) setenv() calls in workflow 12861.0 step2 no longer shows setenv() calls called in the framework's parallel section.

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

Good question if this should be backported. The race condition exists in earlier releases, but we haven't seen crash reports from production. Maybe 14_1_X and 14_0_X could still be useful?

[cmsbuild]

cms-bot internal usage

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-46065/41854

[cmsbuild]

A new Pull Request was created by @makortel for master.

It involves the following packages:

• DQM/DTMonitorClient (dqm)
• L1Trigger/L1CaloTrigger (l1, upgrade)
• L1Trigger/L1THGCal (l1, upgrade)
• L1Trigger/Phase2L1ParticleFlow (l1, upgrade)
• PhysicsTools/TensorFlow (ml)
• RecoEcal/EgammaCoreTools (reconstruction)
• RecoMuon/TrackerSeedGenerator (reconstruction)
• RecoTauTag/HLTProducers (hlt)

@Martin-Grunewald, @aloeliger, @antoniovagnerini, @cmsbuild, @epalencia, @jfernan2, @mandrenguyen, @mmusich, @nothingface0, @rvenditti, @srimanob, @subirsarkar, @syuvivida, @tjavaid, @valsdav, @y19y19 can you please review it and eventually sign? Thanks.
@CeliaFernandez, @Fedespring, @HuguesBrun, @Martin-Grunewald, @Prasant1993, @ReyerBand, @Sam-Harper, @a-kapoor, @abbiendi, @afiqaize, @amarini, @andrea21z, @argiro, @azotz, @battibass, @bellan, @cericeci, @jainshilpi, @jbsauvan, @jhgoh, @lgray, @mbluj, @missirol, @mmusich, @ram1123, @rchatter, @riga, @rociovilar, @sameasy, @silviodonato, @sobhatta, @thomreis, @trocino, @valsdav, @varuns23, @wang0jin this is something you requested to watch as well.
@antoniovilela, @mandrenguyen, @rappoccio, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

[makortel]

@cmsbuild, please test

[cmsbuild]

+1

Size: This PR adds an extra 84KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-407fec/41643/summary.html
COMMIT: bafd04e
CMSSW: CMSSW_14_2_X_2024-09-19-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/46065/41643/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially added 15 lines to the logs
• Reco comparison results: 19 differences found in the comparisons
• DQMHistoTests: Total files compared: 44
• DQMHistoTests: Total histograms compared: 3331158
• DQMHistoTests: Total failures: 2008
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3329130
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 43 files compared)
• Checked 193 log files, 163 edm output root files, 44 DQM output files
• TriggerResults: no differences found

[mmusich]

This PR should probably be accompanied with a PR to cmsdist setting TF_CPP_MIN_LOG_LEVEL=3 in the Tensorflow toolfile.

has this already happened?

[mmusich]

@kandrosov FYI

[mmusich]

+hlt

[valsdav]

+ml

Thanks for the fix.

[makortel]

This PR should probably be accompanied with a PR to cmsdist setting TF_CPP_MIN_LOG_LEVEL=3 in the Tensorflow toolfile.

has this already happened?

Now in cms-sw/cmsdist#9418

[makortel]

@cmsbuild, please test with cms-sw/cmsdist#9418

[makortel]

could you also maybe initiate the backports to 14_1_X and 14_0_X? This way we can issue a test of the PR on the current playback release 14_0_X.

I can make the backports after the next round of tests succeed.

[smuzaffar]

@cmsbuild, please test with cms-sw/cmsdist#9418

@makortel , I have updated cms-sw/cmsdist#9418 to make TF_CPP_MIN_LOG_LEVEL a runtime variable

[cmsbuild]

+1

Size: This PR adds an extra 12KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-407fec/41670/summary.html
COMMIT: bafd04e
CMSSW: CMSSW_14_2_X_2024-09-20-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/46065/41670/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

• @cms-sw Update tag for DataFormats-L1TGlobal to V00-02-00 cmsdist#9417
• @cms-sw Update tag for DataFormats-DetId to V00-02-00 cmsdist#9419
• @cms-sw Update tag for DataFormats-SiStripCluster to V00-02-00 cmsdist#9421
• @cms-sw Update tag for DataFormats-HLTReco to V00-02-00 cmsdist#9420
• @cms-sw Update tag for DataFormats-L1Scouting to V00-04-00 cmsdist#9423
• @cms-sw Update tag for DataFormats-Scouting to V00-04-00 cmsdist#9422
• @cms-sw Update tag for DataFormats-FEDRawData to V00-02-00 cmsdist#9425
• @cms-sw Update tag for DataFormats-Common to V00-02-00 cmsdist#9424
• @cms-sw Update tag for DataFormats-L1ScoutingRawData to V00-03-00 cmsdist#9426

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-407fec/41670/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-407fec/41670/git-merge-result

Comparison Summary

Summary:

• You potentially removed 110 lines from the logs
• Reco comparison results: 80 differences found in the comparisons
• DQMHistoTests: Total files compared: 44
• DQMHistoTests: Total histograms compared: 3331158
• DQMHistoTests: Total failures: 4599
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3326539
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 43 files compared)
• Checked 193 log files, 163 edm output root files, 44 DQM output files
• TriggerResults: no differences found

[makortel]

This PR + cms-sw/cmsdist#9418 seems to remove these printouts

To enable the following instructions: AVX AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.

[antoniovagnerini]

+1

• spurious differences seen from unstable WF and 12834.7 (open issue spurious differences in outputs of Run-3 wfs *.7 #39803 )

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @sextonkennedy, @antoniovilela, @rappoccio, @mandrenguyen (and backports should be raised in the release meeting by the corresponding L2)

[makortel]

could you also maybe initiate the backports to 14_1_X and 14_0_X? This way we can issue a test of the PR on the current playback release 14_0_X.

I can make the backports after the next round of tests succeed.

The backports are in

• 14_1_X: [14_1_X] Remove tensorflow::setLogging() as thread-unsafe #46106 + [14_1_X] Set Tensorflow logging level to 3 cmsdist#9431
• 14_0_X: [14_0_X] Remove tensorflow::setLogging() as thread-unsafe #46107 + [14_0_X] Set Tensorflow logging level to 3 cmsdist#9432

[mandrenguyen]

+1