gemChamberVFATBX0GENeg11 with duplicate labels makes T0 crash on 12_5 reply

[rvenditti]

As pointed in dmwm/T0#4774, the reply on 12_5_1 failed (at merging DQMIO step?) because of duplicate labels in gemChamberVFATBX0GENeg11 histogram.
Error:
----- Begin Fatal Exception 31-Oct-2022 15:01:26 CET-----------------------
An exception of category 'FatalRootError' occurred while
[0] Calling InputSource::readRun_
Additional Info:
[a] Fatal Root Error: @sub=TH1Merger::CheckForDuplicateLabels
Histogram gemChamberVFATBX0GEPos11 has duplicate labels in the x axis. Bin contents will be merged in a single bin

----- End Fatal Exception -------------------------------------------------

Files to reproduce the error are in: /afs/cern.ch/user/c/cmst0/public/PausedJobs/HIReplay2022/job_244/job/WMTaskSpace/cmsRun1

[cmsbuild]

A new Issue was created by @rvenditti .

@Dr15Jones, @perrotta, @dpiparo, @rappoccio, @makortel, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[rvenditti]

can @cms-sw/gem-dpg-l2 kindly have a look?

[makortel]

assign dqm

[cmsbuild]

New categories assigned: dqm

@jfernan2,@ahmad3213,@micsucmed,@rvenditti,@emanueleusai,@syuvivida,@pmandrik you have been requested to review this Pull request/Issue and eventually sign? Thanks

[mmusich]

Looks like the problematic ME is actually in the L1T stack:

cmssw/DQM/L1TMonitor/src/L1TStage2EMTF.cc

Lines 403 to 427 in 7f0cfad

	gemChamberVFATBX[hist][bx - 1] =
	ibooker.book2D("gemChamberVFATBX" + bx_string + name,
	"GEM Chamber vs VFAT at BX = " + std::to_string(bx - 4) + ", " + label,
	42,
	1,
	43,
	24,
	0,
	24); // 8* (0-2) phi part + (0-7) eta part
	gemChamberVFATBX[hist][bx - 1]->setAxisTitle("Chamber, " + label, 1);
	gemChamberVFATBX[hist][bx - 1]->setAxisTitle("VFAT #", 2);
	for (int bin = 1; bin <= 24; bin++)
	gemChamberVFATBX[hist][bx - 1]->setBinLabel(bin, std::to_string(bin - 1), 2);
	int count = 0;
	for (int bin = 1; bin <= 42; ++bin) {
	gemChamberVFATBX[hist][bx - 1]->setBinLabel(bin, std::to_string(bin - count), 1);
	if (bin == 2 || bin == 9 || bin == 16 || bin == 23 || bin == 30 || bin == 37) {
	++bin;
	++count;
	gemChamberVFATBX[hist][bx - 1]->setBinLabel(bin, "N", 1);
	}
	}
	}

adding to the discussion the original author @nickh2000 of PR #38818 (@eyigitba FYI)

[eyigitba]

@mmusich , thanks for notifying me. Does this crash happen in the merging jobs? Could it be due to a missing line such as this one not being set for that one histogram?

gemHitTimingFrac[hist]->getTH2F()->GetXaxis()->SetCanExtend(false)

[mmusich]

hi @eyigitba

Does this crash happen in the merging jobs? Could it be due to a missing line such as this one not being set for that one histogram?

as far as I understand the underlying ROOT version in 12.5.X really doesn't like to have a histogram with two bin labels containing the same name (for reasons that admittedly I don't understand , and @cms-sw/core-l2 might shed some light upon).
You can see similar occurence of this problem fixed here: #39853 (see also issue #39737)

[mmusich]

for reasons that admittedly I don't understand

I've found this old PR #25569 and in the descriptions there's written (quoting)

In the newest version of ROOT, histograms with named bins are
merged via the bin's name, not the index. This means we must
avoid bins having the same bin names.

[nickh2000]

Hi @mmusich,

If you agree, we can proceed by changing these duplicate labels and submitting a backported PR.

How do you suggest we test this change?

(fyi @eyigitba )

[mmusich]

If you agree, we can proceed by changing these duplicate labels and submitting a backported PR.

yes, sure.

How do you suggest we test this change?

in the issue description there's a pointer to the failing job from tier-0: /afs/cern.ch/user/c/cmst0/public/PausedJobs/HIReplay2022/job_244/job/WMTaskSpace/cmsRun1.
If you run PSet.py that you can find in that folder, you should be able to test the fix (without it, it crashes).

[nickh2000]

HI @mmusich,

Thank you for your advice. How do I recreate new DQMIO files for the merging process?

[mmusich]

How do I recreate new DQMIO files for the merging process?

something like:

cmsrel CMSSW_12_5_1 
cd CMSSW_12_5_1/src
cmsenv 
git cms-addpkg Configuration/DataProcessing
python3 Configuration/DataProcessing/test/RunExpressProcessing.py --nThreads 8 --scenario ppEra_Run3 --global-tag 124X_dataRun3_Express_v6  --lfn  <here put some input RAW data> --fevt --dqmio 

should create the DQMIO files.
Notice that I haven't looked to where to find the RAW data for inputing the wf, but I guess @rvenditti or @francescobrivio @germanfgv can help you with that.

[francescobrivio]

@cms-sw/orp-l2 this is the issue that needs to be fixed in DQM before deploying 12_5_X in Tier0

[mmusich]

A possibile solution is proposed in #39952

[perrotta]

urgent
(marking urgent the issues affecting T0 replays)

[emanueleusai]

+dqm

[cmsbuild]

This issue is fully signed and ready to be closed.

[mmusich]

please close