Spurious failing comparisons entries in PR tests for MuonCSCDigisV/CSCDigiTask/CLCT

[perrotta]

Since some while, spurious failing comparisons entries show up in PR tests for MuonCSCDigisV/CSCDigiTask/CLCT.
They are not reproducible (repeated tests may show different comparisons).
A few examples can be seen in the comparisons of PRs #37260 (for wf 10042), #37263 (for wf 1330), ...

In all plots that I have inspected the visible bins looks identical. and therefore the difference could be in the overflow/underflow bins (just guessing, since the number of entries in those plots are larger than the number of entries in the visible bins).

Looking at the recently merged PRs, #37188 ("Make both ME1a and ME1b LCT (A/CLCT) sorted into ring 1") could be the origin of the issue, even if I can't see why. Let me point out that #37188 was also merged in 12_2 and 12_3: if any fix is needed, it should also get backported there.

@sifuluo @jfernan2 could you please have a look?

[cmsbuild]

A new Issue was created by @perrotta Andrea Perrotta.

@Dr15Jones, @perrotta, @dpiparo, @makortel, @smuzaffar, @qliphy can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[perrotta]

assign dqm

[cmsbuild]

New categories assigned: dqm

@jfernan2,@ahmad3213,@rvenditti,@emanueleusai,@pbo0,@pmandrik you have been requested to review this Pull request/Issue and eventually sign? Thanks

[jfernan2]

@barvic please also have a look as CSC Validation developer. Thanks

[sifuluo]

I've found the direct cause for this issue. And we will have a discussion to pick a solution to it asap.

[perrotta]

Supposedly fi*ed by #37282.
Let see if really the failing comparisons disappear from forthcoming PR tests before finally closing this issue

[perrotta]

If the fix works as expected, two backports to 12_3_X and 12_2_X should also be prepared, because the bug is also present in those two release cycles

[jfernan2]

@sifuluo could you please make the backports? Thanks

[perrotta]

@sifuluo @jfernan2 I am observing that in some PR tests based on CMSSW_12_4_X_2022-03-22-1100, which already includes #37282, the spurious differences in the DQM comparisons for MuonCSCDigisV/CSCDigiTask/CLCT are still there, see for example wf 10042.0 in #37265 (comment)

[sifuluo]

Let's see if this one solves it

[perrotta]

Ok, #37319 looks like it has fixed the issue, or at least I did not see those failing comparisons any more in the PR tests since it was merged in master in CMSSW_12_4_X_2022-03-24-1100

Now backports of the combined fix, #37282 + #37319, should be prepared for:

• CMSSW_12_2_X (to fix the bug introduced with Make both ME1a and ME1b LCT (A/CLCT) sorted into ring 1. #37179)
• CMSSW_12_3_X (to fix the bug introduced with Make both ME1a and ME1b LCT (A/CLCT) sorted into ring 1. #37217)

because the bug is also present in those two release cycles

@sifuluo could you please take care of it? Please notice that a new 12_2_X release is expected to be built in short, and it would be a pity not to include these fixes in it.

[perrotta]

+1

• Fixed by Fix resolution validation plots after moving ME1a CLCT from ring 4 to ring 1 #37282
  and ME1a CLCT will use ME1a SimHit Strip and be saved into ME1a plot #37319, and their backports to 12_2_X and 12_3_X