GEM Det ID error in Online and Offline DQM

[rvenditti]

We observed the following error message:
Exception Message:
GEMDetId ctor: Invalid parameters: region 1 ring 1 station 1 layer 0 chamber 5 ieta -99
in:

• Online DQM, in clients l1tstage2 and l1tstage2emulator l1stage2emulog.txt
• Express reconstruction: https://cms-talk.web.cern.ch/t/paused-jobs-for-express-run359045-streamexpress/15232

The error seems to be produced by:

cmssw/DataFormats/MuonDetId/interface/GEMDetId.h

Lines 100 to 102 in 6d2f660

	throw cms::Exception("InvalidDetId")
	<< "GEMDetId ctor: Invalid parameters: region " << region << " ring " << ring << " station " << station
	<< " layer " << layer << " chamber " << chamber << " ieta " << ieta << std::endl;

First investigations:

• Express reco does not give error in 12_4_8, but it crashes with 12_4_8+PR39389 and 12_4_9 (thanks @francescobrivio for checking)
• Online system does not crash with 12_4_8, but it crashes with 12_4_8+PR39389 and 12_4_9 (thanks @syuvivida for checking)

[cmsbuild]

A new Issue was created by @rvenditti .

@Dr15Jones, @perrotta, @dpiparo, @rappoccio, @makortel, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[francescobrivio]

Just to add a quick recipe for reproducing the Express crash:
12_4_9 - crash:

cmsrel CMSSW_12_4_9
cd CMSSW_12_4_9/src
cmsenv
cp /afs/cern.ch/user/c/cmst0/public/PausedJobs/Run2022E/InvalidDetIdError/job_1924/Express-5bfd16d5-50da-4ab0-9650-a2f3dae3bc11-3-logArchive.tar.gz .
tar -zxvf Express-5bfd16d5-50da-4ab0-9650-a2f3dae3bc11-3-logArchive.tar.gz 
cd job/WMTaskSpace/cmsRun1
cmsRun PSet.py

12_4_8 - no crash:

cmsrel CMSSW_12_4_8
cd CMSSW_12_4_8/src
cmsenv
cp /afs/cern.ch/user/c/cmst0/public/PausedJobs/Run2022E/InvalidDetIdError/job_1924/Express-5bfd16d5-50da-4ab0-9650-a2f3dae3bc11-3-logArchive.tar.gz .
tar -zxvf Express-5bfd16d5-50da-4ab0-9650-a2f3dae3bc11-3-logArchive.tar.gz 
cd job/WMTaskSpace/cmsRun1
cp /afs/cern.ch/work/f/fbrivio/public/tier0_issue_run359045/my_config.py .
cmsRun my_config.py

Note1: in 12_4_8 you need to copy the my_config.py where I disabled ParticleNetJetTagMonitor and SiPixelCalSingleMuonAnalyzer which were introduced in 12_4_9.

Note2: to introduce the crash in 12_4_8 just do:

cd CMSSW_12_4_8/src
cmsenv
git cms-addpkg EventFilter/L1TRawToDigi
git cherry-pick 65ea079ee92db467fbe026058220aa78fa0a7cf0
git cherry-pick 195accc73a10b8749fa1a6fd3fd15d6687afd5ce
scram b -j 8

and then again run my_config.py

[francescobrivio]

Let me add from the @cms-sw/alca-l2 point of view:

• In CMSSW_12_4_9 the GTs used have not changed wrt 12_4_8 (see https://twiki.cern.ch/twiki/bin/view/CMS/LatestOnlineGTs)
• I have checked the Express GT and the GEM geometry tags haven't been updated since July 2022

@eyigitba you are the author of the original PR: any idea?

[francescobrivio]

let me tag also @hyunyong @quark2 @jshlee @watson-ij as GEM experts

[rappoccio]

urgent

[makortel]

assign dqm

[cmsbuild]

New categories assigned: dqm

@jfernan2,@ahmad3213,@micsucmed,@rvenditti,@emanueleusai,@syuvivida,@pmandrik you have been requested to review this Pull request/Issue and eventually sign? Thanks

[makortel]

assign l1

[cmsbuild]

New categories assigned: l1

@epalencia,@rekovic,@cecilecaillol you have been requested to review this Pull request/Issue and eventually sign? Thanks

[francescobrivio]

To add even more info:
as pointed out by @davidlange6 the run used in the Tier0 replay (https://cmsoms.cern.ch/cms/runs/report?cms_run=356824&cms_run_sequence=GLOBAL-RUN) did not have GEM included in the GlobalRun, that's probably why the crash did not appear in the replay...

[eyigitba]

Hi, I think I saw this error in some other workflow. I don't why this wasn't caught in my tests for this PR. I'll check and submit a PR to fix this today.

[eyigitba]

I found the bug, it was a stupid mistake by me. It exists in all 3 PRs. I'll submit fixes now. Sorry about that.

[perrotta]

@eyigitba prepared the fix, in #39460, backported as #39461 in 12_5_X and #39462 in 12_4_X

The fix is quite logical, therefore it must be applied. Can anybody please verify whether it fully fixes the issue, and there are no other similar bugs hidden somewhere else in the code?

[francescobrivio]

I'm testing it offline with the recipe I provided above, but it's better if it gets tested in DQM online (@cms-sw/dqm-l2) as well. I'm not sure if can be tested in Tier0 without a release...

[syuvivida]

I am building and testing PR 39462 now, will let you know the results

[francescobrivio]

Ok I've re-run the tarball posted by Tier0 experts for the problematic job:

• plain CMSSW_12_4_9 --> An exception of category 'InvalidDetId' occurred
• CMSSW_12_4_9 + 7ebacc9 --> no crash

So the fix seems correct at least from the offline point of view.

[syuvivida]

We tested PR #39462 at the playback using the crashed run 359045 (reported by the DQM group earlier). The run finished OK without errors (25 LS tested). The test was done with CMSSW_12_4_9 + PR 39462.

[perrotta]

fixed by #39460, backported as #39461 in 12_5_X and #39462 in 12_4_X

[perrotta]

@rvenditti I think you can close this issue