Spurious comparison differences in workflows 9.0 and 25.0

[makortel]

Workflow 9.0 seems to show occasional differences in 13 histograms in GlobalHitsV/MCGeant folder. I first saw this in tests of #43338 (comment) and #43338 (comment). The histograms in question are

• hGeantVtxMulti
• hGeantTrkE
• hGeantTrkPt
• hGeantVtxPhi
• hGeantVtxX1
• hGeantVtxX2
• hGeantVtxZ1
• hGeantVtxRad1
• hGeantVtxZ2
• hGeantVtxRad2
• hGeantVtxY1
• hGeantVtxY2
• hGeantVtxEta

Example:

[makortel]

assign simulation, dqm

[cmsbuild]

New categories assigned: simulation,dqm

@civanch,@mdhildreth,@rvenditti,@syuvivida,@tjavaid,@nothingface0,@antoniovagnerini you have been requested to review this Pull request/Issue and eventually sign? Thanks

[cmsbuild]

A new Issue was created by @makortel Matti Kortelainen.

@Dr15Jones, @sextonkennedy, @rappoccio, @antoniovilela, @makortel, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[makortel]

Poking around a bit of recent PRs

I have seen these difference to occur in

• CMSSW_14_0_X_2023-11-20-2300 (ran on Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz)
• CMSSW_14_0_X_2023-11-24-1100 (ran on Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz)
• CMSSW_14_0_X_2023-11-24-2300 (ran on Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz)

but not in

• CMSSW_14_0_X_2023-11-20-1100 (ran on Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz)
• CMSSW_14_0_X_2023-11-22-2300
• CMSSW_14_0_X_2023-11-26-2300
• CMSSW_14_0_X_2023-11-27-1100 (ran on Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz)

Although now I saw also one PR tests in CMSSW_14_0_X_2023-11-24-1100 that did not show these differences, so maybe the cause is not specific to specific IBs.

[makortel]

A related issue #34448

[makortel]

Tests in #43386 (comment) show similar differences in workflow 25.0