HLT reproducibility issues after integration of #44415

[mmusich]

During IB testing of the latest PRef menus (/dev/CMSSW_14_0_0/PRef/V72), the hltIntegrationTests complain about differences in trigger results when (1) running the full menu vs (2) running each Path standalone for the following paths:

• HLT_PPRefDmesonTrackingGlobal_Dpt25_v3
• HLT_PPRefDmesonTrackingGlobal_Dpt35_v3
• HLT_PPRefDmesonTrackingGlobal_Dpt45_v3
• HLT_PPRefDmesonTrackingGlobal_Dpt60_v3

see e.g.:

• here for CMSSW_14_1_X
• here for CMSSW_14_0_X

Facts:

• this issue does not happen with the status of the HLTrigger/Configuration prior to integration of HLT menu development for 14_0_X (4/N) [14_1_X] #44415;
• it has been seen in multiple release cycles (14_(0|1)_X) when manually testing PRef/V72 + customizeHLTforCMSSW;
• there is no runtime error or log-warnings from cmsRun, the issue so far is purely in the reproducibility of HLT outputs;

Reproducer (tested in CMSSW_14_1_X_2024-03-16-1100):

#!/bin/bash

jobTag=threads4
hltMenu=/dev/CMSSW_14_0_0/PRef/V72

check_log () {
  grep '0 HLT_PPRefDmesonTrackingGlobal_Dpt60_v' $1 | grep TrigReport
}

run(){
  echo $2
  cp $1 $2.py
  cat <<EOF >> $2.py

process.options.numberOfThreads = 4
process.options.numberOfStreams = 0

process.hltOutputMinimal.outputCommands += [
  'keep *_hltFullIter*_*_*',
]

process.hltOutputMinimal.fileName = '${2}.root'
EOF
  cmsRun "${2}".py &> "${2}".log
  check_log "${2}".log
}

hltGetCmd="hltGetConfiguration ${hltMenu}"
hltGetCmd+=" --globaltag auto:run3_mc_PRef --mc --unprescale --output minimal --max-events -1"
hltGetCmd+=" --input /store/group/dpg_trigger/comm_trigger/TriggerStudiesGroup/STORM/debug/170324_reproIssueWithPRef/RelVal_DigiL1Raw_PRef_MC.root"

#echo $hltGetCmd

configLabel=hlt_"${jobTag}"_onlyPPRefDmesonTrackingGlobal
#echo "${configLabel}".py
${hltGetCmd} --paths HLT_PPRefDmesonTrackingGlobal_Dpt60_v3 > "${configLabel}".py
for job_i in {0..9}; do run "${configLabel}".py "${configLabel}"_"${job_i}"; done; unset job_i;

configLabel=hlt_"${jobTag}"_full
${hltGetCmd} > "${configLabel}".py
for job_i in {0..9}; do run "${configLabel}".py "${configLabel}"_"${job_i}"; done; unset job_i;

results obtained:

hlt_threads4_onlyPPRefDmesonTrackingGlobal_0
TrigReport     1    0        100          4         96          0 HLT_PPRefDmesonTrackingGlobal_Dpt60_v3
TrigReport        100        100        100          0          0 HLT_PPRefDmesonTrackingGlobal_Dpt60_v3
hlt_threads4_onlyPPRefDmesonTrackingGlobal_1
TrigReport     1    0        100          4         96          0 HLT_PPRefDmesonTrackingGlobal_Dpt60_v3
TrigReport        100        100        100          0          0 HLT_PPRefDmesonTrackingGlobal_Dpt60_v3
hlt_threads4_onlyPPRefDmesonTrackingGlobal_2
TrigReport     1    0        100          4         96          0 HLT_PPRefDmesonTrackingGlobal_Dpt60_v3
TrigReport        100        100        100          0          0 HLT_PPRefDmesonTrackingGlobal_Dpt60_v3
hlt_threads4_onlyPPRefDmesonTrackingGlobal_3
TrigReport     1    0        100          4         96          0 HLT_PPRefDmesonTrackingGlobal_Dpt60_v3
TrigReport        100        100        100          0          0 HLT_PPRefDmesonTrackingGlobal_Dpt60_v3
hlt_threads4_onlyPPRefDmesonTrackingGlobal_4
TrigReport     1    0        100          4         96          0 HLT_PPRefDmesonTrackingGlobal_Dpt60_v3
TrigReport        100        100        100          0          0 HLT_PPRefDmesonTrackingGlobal_Dpt60_v3
hlt_threads4_onlyPPRefDmesonTrackingGlobal_5
TrigReport     1    0        100          4         96          0 HLT_PPRefDmesonTrackingGlobal_Dpt60_v3
TrigReport        100        100        100          0          0 HLT_PPRefDmesonTrackingGlobal_Dpt60_v3
hlt_threads4_onlyPPRefDmesonTrackingGlobal_6
TrigReport     1    0        100          4         96          0 HLT_PPRefDmesonTrackingGlobal_Dpt60_v3
TrigReport        100        100        100          0          0 HLT_PPRefDmesonTrackingGlobal_Dpt60_v3
hlt_threads4_onlyPPRefDmesonTrackingGlobal_7
TrigReport     1    0        100          4         96          0 HLT_PPRefDmesonTrackingGlobal_Dpt60_v3
TrigReport        100        100        100          0          0 HLT_PPRefDmesonTrackingGlobal_Dpt60_v3
hlt_threads4_onlyPPRefDmesonTrackingGlobal_8
TrigReport     1    0        100          4         96          0 HLT_PPRefDmesonTrackingGlobal_Dpt60_v3
TrigReport        100        100        100          0          0 HLT_PPRefDmesonTrackingGlobal_Dpt60_v3
hlt_threads4_onlyPPRefDmesonTrackingGlobal_9
TrigReport     1    0        100          4         96          0 HLT_PPRefDmesonTrackingGlobal_Dpt60_v3
TrigReport        100        100        100          0          0 HLT_PPRefDmesonTrackingGlobal_Dpt60_v3
hlt_threads4_full_0
TrigReport     1  106        100          5         95          0 HLT_PPRefDmesonTrackingGlobal_Dpt60_v3
TrigReport        100        100        100          0          0 HLT_PPRefDmesonTrackingGlobal_Dpt60_v3
hlt_threads4_full_1
TrigReport     1  106        100          5         95          0 HLT_PPRefDmesonTrackingGlobal_Dpt60_v3
TrigReport        100        100        100          0          0 HLT_PPRefDmesonTrackingGlobal_Dpt60_v3
hlt_threads4_full_2
TrigReport     1  106        100          5         95          0 HLT_PPRefDmesonTrackingGlobal_Dpt60_v3
TrigReport        100        100        100          0          0 HLT_PPRefDmesonTrackingGlobal_Dpt60_v3
hlt_threads4_full_3
TrigReport     1  106        100          5         95          0 HLT_PPRefDmesonTrackingGlobal_Dpt60_v3
TrigReport        100        100        100          0          0 HLT_PPRefDmesonTrackingGlobal_Dpt60_v3
hlt_threads4_full_4
TrigReport     1  106        100          5         95          0 HLT_PPRefDmesonTrackingGlobal_Dpt60_v3
TrigReport        100        100        100          0          0 HLT_PPRefDmesonTrackingGlobal_Dpt60_v3
hlt_threads4_full_5
TrigReport     1  106        100          5         95          0 HLT_PPRefDmesonTrackingGlobal_Dpt60_v3
TrigReport        100        100        100          0          0 HLT_PPRefDmesonTrackingGlobal_Dpt60_v3
hlt_threads4_full_6
TrigReport     1  106        100          5         95          0 HLT_PPRefDmesonTrackingGlobal_Dpt60_v3
TrigReport        100        100        100          0          0 HLT_PPRefDmesonTrackingGlobal_Dpt60_v3
hlt_threads4_full_7
TrigReport     1  106        100          5         95          0 HLT_PPRefDmesonTrackingGlobal_Dpt60_v3
TrigReport        100        100        100          0          0 HLT_PPRefDmesonTrackingGlobal_Dpt60_v3
hlt_threads4_full_8
TrigReport     1  106        100          5         95          0 HLT_PPRefDmesonTrackingGlobal_Dpt60_v3
TrigReport        100        100        100          0          0 HLT_PPRefDmesonTrackingGlobal_Dpt60_v3
hlt_threads4_full_9
TrigReport     1  106        100          5         95          0 HLT_PPRefDmesonTrackingGlobal_Dpt60_v3
TrigReport        100        100        100          0          0 HLT_PPRefDmesonTrackingGlobal_Dpt60_v3

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @mmusich.

@Dr15Jones, @antoniovilela, @rappoccio, @smuzaffar, @sextonkennedy, @makortel can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[mmusich]

assign hlt

[cmsbuild]

New categories assigned: hlt

@Martin-Grunewald,@mmusich you have been requested to review this Pull request/Issue and eventually sign? Thanks

[mmusich]

tracked in the HLT JIRA: https://its.cern.ch/jira/browse/CMSHLT-3116

[mmusich]

+hlt

The following paths:

• HLT_PPRefDmesonTrackingGlobal_Dpt25_v*
• HLT_PPRefDmesonTrackingGlobal_Dpt35_v*
• HLT_PPRefDmesonTrackingGlobal_Dpt45_v*
• HLT_PPRefDmesonTrackingGlobal_Dpt60_v*

have been removed from the PRef HLT table via CMSHLT-3146.
This ticket entered the cmssw incarnation of the HLT menus in PRs #44717 (master, CMSS_14_1_X) and #44718 (CMSSW_14_0_X).
As expected, after the integration of those PR there hasn't been any failures in HLT IB tests.
The ticket CMSHLT-3116 remains open to track the issue for further investigation.
If / when these paths will be re-inserted in the PRef table they will need to be demonstrably reproducible as per TSG policy.

[cmsbuild]

This issue is fully signed and ready to be closed.

[makortel]

@cmsbuild, please close