[DBG] Exception of category Invalid DetId in RelVal wf 23234.0

[aandvalenzuela]

Hello,

RelVal wf 23234.0 fails in DBG IBs with an exception of category Invalid DetId. Here the log:

DIGI:pdigi_valid,L1TrackTrigger,L1,DIGI2RAW,HLT:@fake2,ENDJOB
We have determined that this is simulation (if not, rerun cmsDriver.py with --data)
entry file:step1.root
Step: DIGI Spec: ['pdigi_valid']
Step: L1TrackTrigger Spec:
Step: L1 Spec:
Step: DIGI2RAW Spec:
Step: HLT Spec: ['@fake2']
Step: ENDJOB Spec:
customising the process with customizeHLTforMC from HLTrigger/Configuration/customizeHLTforMC
Starting  cmsRun  step2_DIGI_L1TrackTrigger_L1_DIGI2RAW_HLT.py
%MSG-i ThreadStreamSetup:  (NoModuleName) 17-Mar-2023 14:06:37 CET pre-events
setting # threads 4
setting # streams 4
%MSG
17-Mar-2023 14:06:50 CET  Initiating request to open file file:step1.root
17-Mar-2023 14:06:51 CET  Successfully opened file file:step1.root
----- Begin Fatal Exception 17-Mar-2023 14:08:53 CET-----------------------
An exception of category 'Invalid DetId' occurred while
   [0] Calling method for module HGCalBackendLayer1Producer/'l1tHGCalBackEndLayer1Producer'
   [1] Prefetching for EventSetup module HGCalTriggerGeometryESProducer/'l1tHGCalTriggerGeometryESProducer'
   [2] Calling method for EventSetup module HGCalGeometryESProducer/'HFNoseGeometryESProducer'
Exception Message:
Cannot initialize HGCSiliconDetId from 6c8bd800
----- End Fatal Exception -------------------------------------------------
----- Begin Fatal Exception 17-Mar-2023 14:08:53 CET-----------------------
An exception of category 'Invalid DetId' occurred while
   [0] Calling method for module HGCalBackendLayer1Producer/'l1tHGCalBackEndLayer1Producer'
   [1] Prefetching for EventSetup module HGCalTriggerGeometryESProducer/'l1tHGCalTriggerGeometryESProducer'
   [2] Calling method for EventSetup module HGCalGeometryESProducer/'HFNoseGeometryESProducer'
Exception Message:
Cannot initialize HGCSiliconDetId from 6c8bd800
----- End Fatal Exception -------------------------------------------------
----- Begin Fatal Exception 17-Mar-2023 14:08:53 CET-----------------------
An exception of category 'Invalid DetId' occurred while
   [0] Calling method for module HGCalBackendLayer1Producer/'l1tHGCalBackEndLayer1Producer'
   [1] Prefetching for EventSetup module HGCalTriggerGeometryESProducer/'l1tHGCalTriggerGeometryESProducer'
   [2] Calling method for EventSetup module HGCalGeometryESProducer/'HFNoseGeometryESProducer'
Exception Message:
Cannot initialize HGCSiliconDetId from 6c8bd800
----- End Fatal Exception -------------------------------------------------
----- Begin Fatal Exception 17-Mar-2023 14:08:53 CET-----------------------
An exception of category 'Invalid DetId' occurred while
   [0] Calling method for module HGCalBackendLayer1Producer/'l1tHGCalBackEndLayer1Producer'
   [1] Prefetching for EventSetup module HGCalTriggerGeometryESProducer/'l1tHGCalTriggerGeometryESProducer'
   [2] Calling method for EventSetup module HGCalGeometryESProducer/'HFNoseGeometryESProducer'
Exception Message:
Cannot initialize HGCSiliconDetId from 6c8bd800
----- End Fatal Exception -------------------------------------------------
17-Mar-2023 14:08:53 CET  Closed file file:step1.root
MessageLogger: dropped waiting message count 6944

In the latest DBG IB, the test has timed out. But I have managed to reproduce the Invalid DetId exception in one of our cmsdev machines, which we saw for the first time in CMSSW_13_1_X-2023-03-09-2300.

Thanks!

[cmsbuild]

A new Issue was created by @aandvalenzuela Andrea Valenzuela.

@Dr15Jones, @perrotta, @dpiparo, @rappoccio, @makortel, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[makortel]

assign geometry, upgrade

[makortel]

FYI @cms-sw/hgcal-dpg-l2

[cmsbuild]

New categories assigned: geometry,upgrade

@mdhildreth,@AdrianoDee,@Dr15Jones,@srimanob,@makortel,@bsunanda,@civanch you have been requested to review this Pull request/Issue and eventually sign? Thanks

[iarspider]

@cms-sw/hgcal-dpg-l2 gentle ping

[srimanob]

For my education, what is different between DBG and standard IB? I see that the workflow runs fine in standard IB. I assume if it is geometry issue, we should fail in both.

[iarspider]

For my education, what is different between DBG and standard IB? I see that the workflow runs fine in standard IB. I assume if it is geometry issue, we should fail in both.

I believe in DBG_X builds we enable extra logging (both via logger configuration and defining some flags like EDM_ML_DEBUG)

[srimanob]

Thanks. I also see that you can reproduce it with normal release.

@bsunanda FYI

[iarspider]

@bsunanda gentle ping on this issue.

[iarspider]

Failed again in CMSSW_13_3_DBG_X_2023-10-26-2300 - @bsunanda @srimanob

[aandvalenzuela]

Hello @bsunanda, @srimanob
This issue is still present in CMSSW_13_3_DBG_X_2023-11-02-2300

[bsunanda]

Can you let us know how to get the debug version and which workflow produces this? This is happening for HFNose and Mariarosaria is the author of that portion

…

________________________________ From: Andrea Valenzuela ***@***.***> Sent: 03 November 2023 19:03 To: cms-sw/cmssw ***@***.***> Cc: Sunanda Banerjee ***@***.***>; Mention ***@***.***> Subject: Re: [cms-sw/cmssw] [DBG] Exception of category `Invalid DetId` in RelVal wf `23234.0` (Issue #41088) Hello @bsunanda<https://github.com/bsunanda>, @srimanob<https://github.com/srimanob> This issue is still present in CMSSW_13_3_DBG_X_2023-11-02-2300<https://cmssdt.cern.ch/SDT/cgi-bin/logreader/el8_amd64_gcc12/CMSSW_13_3_DBG_X_2023-11-02-2300/pyRelValMatrixLogs/run/23234.0_TTbar_14TeV+2026D94/step2_TTbar_14TeV+2026D94.log#/> — Reply to this email directly, view it on GitHub<#41088 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABGMZOQS2ZFP7QN6HDQNUXLYCTXBNAVCNFSM6AAAAAAV6VCCJ2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOJSGQ2DIMZWG4>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[iarspider]

on any lxplus or dev machine:

$ cmssw-el8
$ scram -a el8_amd64_gcc11 project CMSSW_13_3_DBG_X_2023-11-02-2300
$ cd CMSSW_13_3_DBG_X_2023-11-02-2300/src
$ cmsenv
$ runTheMatrix.py -i all -t 4 -l 23234.0 --ibeos

[aandvalenzuela]

Thanks @iarspider, I believe the architecture should be el8_amd64_gcc12, which goes on the second command:

scram -a el8_amd64_gcc12 project CMSSW_13_3_DBG_X_2023-11-02-2300