HI Testing - ppEra_Run3_2024_UPC scenario

[LinaresToine]

I am opening this issue to address a problem found while running a HI replay.

The release I'm using is CMSSW_14_1_0

The cmstalk report is: https://cms-talk.web.cern.ch/t/heavy-ion-test-no-dedxcalibrationrcd-record-found-in-the-eventsetup/51252

I believe the error has to do with the scenario: ppEra_Run3_2024_UPC, as only those PDs with this scenario are affected.

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @LinaresToine.

@Dr15Jones, @antoniovilela, @makortel, @mandrenguyen, @rappoccio, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[mmusich]

I think the issue is just that the GT used for the replay (dmwm/T0#4988 I suppose)

https://cms-conddb.cern.ch/cmsDbBrowser/list/Prod/gts/141X_dataRun3_Prompt_v2

does not contain that record.

[mmusich]

assign alca

[cmsbuild]

New categories assigned: alca

@atpathak,@consuegs,@perrotta you have been requested to review this Pull request/Issue and eventually sign? Thanks

[perrotta]

I think the issue is just that the GT used for the replay (dmwm/T0#4988 I suppose)

https://cms-conddb.cern.ch/cmsDbBrowser/list/Prod/gts/141X_dataRun3_Prompt_v2

does not contain that record.

Yes, please use 141X_dataRun3_Prompt_v3 instead

The latest 141X online GTs are:

• 141X_dataRun3_HLT_v1
• 141X_dataRun3_Express_v3
• 141X_dataRun3_Prompt_v3

(see also P9 in the intro of the last AlCaDB meeting, or the GTs in autoCond.py of the HEAD of CMSSW_14_1_X)

[LinaresToine]

Thank you @mmusich and @perrotta for your help. I will try the new GTs

[LinaresToine]

Changing the global tag fixed the previous issue, however, I now see the following error message:

CRITICAL:CMSSW:Error running cmsRun
{'arguments': ['/bin/bash', '/srv/job/WMTaskSpace/cmsRun1/cmsRun1-main.sh', '', 'el8_amd64_gcc12', 'scramv1', 'CMSSW', 'CMSSW_14_1_0', 'FrameworkJobReport.xml', 'cmsRun', 'PSet.py', '']}
CMSSW Return code: 8006

2024-10-01 21:21:56,915:CRITICAL:CMSSW:Error message: An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 374951 lumi: 31 event: 43770934 stream: 0
   [1] Running path 'dqmoffline_9_step'
   [2] Prefetching for module SMPDQM/'SMPDQM'
   [3] Prefetching for module MuonProducer/'muons'
   [4] Prefetching for module MuonIdProducer/'muons1stStep'
   [5] Prefetching for module DuplicateListMerger/'generalTracks'
   [6] Prefetching for module TrackProducer/'mergedDuplicateTracks'
   [7] Prefetching for module DuplicateTrackMerger/'duplicateTrackCandidates'
   [8] Prefetching for module TrackCollectionMerger/'preDuplicateMergingGeneralTracks'
   [9] Prefetching for module TrackCollectionMerger/'earlyGeneralTracks'
   [10] Prefetching for module TrackProducer/'initialStepTracks'
   [11] Prefetching for module MkFitOutputConverter/'initialStepTrackCandidates'
   [12] Prefetching for module MkFitEventOfHitsProducer/'mkFitEventOfHits'
   [13] Prefetching for module MkFitSiPixelHitConverter/'mkFitSiPixelHits'
   [14] Prefetching for module SiPixelRecHitConverter/'siPixelRecHits'
   [15] Prefetching for module JetCoreClusterSplitter/'siPixelClusters'
   [16] Prefetching for module SiPixelClusterProducer/'siPixelClustersPreSplitting@cpu'
   [17] Calling method for module SiPixelRawToDigi/'siPixelDigis@cpu'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for type: FEDRawDataCollection
Looking for module label: rawDataCollector
Looking for productInstanceName: 

   Additional Info:
      [a] If you wish to continue processing events after a ProductNotFound exception,
add "TryToContinue = cms.untracked.vstring('ProductNotFound')" to the "options" PSet in the configuration.

@mmusich any idea?
Should this be a part of a separate issue? Or should I rename this issue to include all cmssw-related HI Testing errors we face?

[mmusich]

@LinaresToine

any idea?

Maybe. On which stream are you getting this error? With which scenario are you processing?
A tarball might be useful.

Should this be a part of a separate issue? Or should I rename this issue to include all cmssw-related HI Testing errors we face?

For the sake of economy you can keep it here, but then please change title and description..

[perrotta]

assign reconstruction

• For EventFilter/SiPixelRawToDigi

[cmsbuild]

New categories assigned: reconstruction

@jfernan2,@mandrenguyen you have been requested to review this Pull request/Issue and eventually sign? Thanks

[mmusich]

assign reconstruction

I think this is premature (and for the moment unnecessary), there seems to be some level of misconfiguration in the tier0 to solve first.

[mmusich]

I think the error at #46186 (comment) is due to a combination of factors.

• several PDs that should have been reconstructed with ppEra_Run3_pp_on_PbPb_2024 are instead reconstructed with ppEra_Run3_2024_UPC (e.g. HIHLTPhysics , HITrackerNZS, HIEmptyBX, etc., see https://github.com/dmwm/T0/pull/4988/files#r1783935965).
• the implementation of ppEra_Run3_2024_UPC meant for HIForward* seems to be missing the repacking flag that instead the scenario used in 2023, see https://github.com/cms-sw/cmssw/pull/45981/files#r1783939492

[mmusich]

unassign reconstruction

[mmusich]

assign operations

[cmsbuild]

New categories assigned: operations

@antoniovilela,@davidlange6,@fabiocos,@mandrenguyen,@rappoccio you have been requested to review this Pull request/Issue and eventually sign? Thanks

[perrotta]

+alca

• The AlCaDB related part of this issue was fixed once updated with the correct GT, see HI Testing - ppEra_Run3_2024_UPC scenario #46186 (comment)

[mandrenguyen]

+1
The "operations" part should be fixed by #46194
Let's open a new issue if there are further problems in the heavy-ion T0 processing configuration.

[cmsbuild]

This issue is fully signed and ready to be closed.