Non-reproducibility in 29634.911

[makortel]

The comparisons of workflow 29634.911 show differences randomly.

[makortel]

assign geometry, upgrade

[cmsbuild]

New categories assigned: geometry,upgrade

@Dr15Jones,@civanch,@bsunanda,@makortel,@mdhildreth,@srimanob,@subirsarkar you have been requested to review this Pull request/Issue and eventually sign? Thanks

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @makortel.

@Dr15Jones, @antoniovilela, @makortel, @mandrenguyen, @rappoccio, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[makortel]

The non-reproducibility is present even after #45442 was merged, I noticed it in #45443 (comment)

[srimanob]

After adding 24834.911 back to the PR test, I see that 24834.911 seems to be stable. In addition, the issues seems to show up only in Tracking, EGamma, BTag, so it seems to me that it relates to geometry, and the issue just comes to the surface when we fix HGCal initialization, and rule out the issue from HGCal failure in PR test.

Issue may come from Tracker part which is also changed between D98 and D110.

[perrotta]

Also observed in #45699

[mmusich]

Recently I've been seeing non-reproducibility in HLT trigger results in workflow 29634.911 and 29634.911 alone.
Recent examples:

• [MISC1] py2/3 compatibility:drop use of __future__ #46778 (comment)
• Fix DQM config test: Increase number of sequences to process #46791 (comment)
• [ALCA-RECONSTRUCTION] py2/3 compatibility:drop use of __future__ #46775 (comment)
• [SIMULATION] py2/3 compatibility:drop use of __future__ #46774 (comment)
• ....

The reproducibility issue is always of the type:

Found 10 matching events, out of which 1 have different HLT results

      Events    Accepted      Gained        Lost       Other  Trigger
          10           4          +1           -           -  pPuppiHT450
          10           3          +1           -           -  pSinglePuppiJet230
          10           0           -           -          ~1  HLT_AK4PFPuppiJet520
          10           0           -           -          ~1  HLT_PFPuppiHT1070
Created the following JSON files:
 /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/upload/triggerResults/29634.911_TTbar_14TeV+Run4D110_DD4hep/HLT.json   

did we change anything recently in the L1T puppi jet seeding algorithm in cmssw ? @cms-sw/l1-l2

[smuzaffar]

@mmusich , today we have enabled the usage of AuthenticAMD nodes for PR/baseline relval comparison (#46669) . Test AuthenticAMD via cms-sw/cms-bot#2374 did show this non-reproducibility 1 out of 7 times

[mmusich]

today we have enabled the usage of AuthenticAMD nodes for PR/baseline relval comparison [...] did show this non-reproducibility 1 out of 7 times

thanks, the question to L1 stands.