[DBG_X] RelVal 140.58 Step 2: Corrupted data in SiStripNoises::getNoise #42162

iarspider · 2023-06-30T08:54:57Z

The following exceptions are reported:

REPACK:DigiToApproxClusterRaw,ENDJOB
We have determined that this is simulation (if not, rerun cmsDriver.py with --data)
entry filelist:step1_dasquery.log
found files:  ['/store/hidata/HIRun2018A/HIHardProbes/RAW/v1/000/326/479/00000/0E2CC5D5-9D87-7348-9219-B00CD718C847.root', '/store/hidata/HIRun2018A/HIHardProbes/RAW/v1/000/326/479/00000/45001EBC-B4D4-9043-A276-8F3AF621C64A.root', '/store/hidata/HIRun2018A/HIHardProbes/RAW/v1/000/326/479/00000/7B3F72ED-E183-3F4B-9FE4-DAE6D911403E.root', '/store/hidata/HIRun2018A/HIHardProbes/RAW/v1/000/326/479/00000/853DBE29-53BA-9A44-9FDD-58E4E9064EB1.root']
Step: REPACK Spec: ['DigiToApproxClusterRaw']
Step: ENDJOB Spec: 
customising the process with customisePostEra_Run2_2018_pp_on_AA from Configuration/DataProcessing/RecoTLR
customising the process with customiseWithTimeMemorySummary from Validation/Performance/TimeMemorySummary
Starting python2 /data/cmsbld/jenkins/workspace/ib-run-relvals/cms-bot/monitor_workflow.py timeout --signal SIGTERM 9000  cmsRun -j JobReport2.xml  step2_REPACK.py
%MSG-i ThreadStreamSetup:  (NoModuleName) 30-Jun-2023 08:32:40 CEST pre-events
setting # threads 4
setting # streams 4
%MSG
30-Jun-2023 08:33:02 CEST  Initiating request to open file root://eoscms.cern.ch//eos/cms/store/user/cmsbuild/store/hidata/HIRun2018A/HIHardProbes/RAW/v1/000/326/479/00000/0E2CC5D5-9D87-7348-9219-B00CD718C847.root
30-Jun-2023 08:33:05 CEST  Successfully opened file root://eoscms.cern.ch//eos/cms/store/user/cmsbuild/store/hidata/HIRun2018A/HIHardProbes/RAW/v1/000/326/479/00000/0E2CC5D5-9D87-7348-9219-B00CD718C847.root
Begin processing the 1st record. Run 326479, Event 1394020, LumiSection 7 on stream 3 at 30-Jun-2023 08:33:13.279 CEST
Begin processing the 2nd record. Run 326479, Event 1579493, LumiSection 7 on stream 2 at 30-Jun-2023 08:33:13.279 CEST
Begin processing the 3rd record. Run 326479, Event 1402087, LumiSection 7 on stream 0 at 30-Jun-2023 08:33:13.279 CEST
Begin processing the 4th record. Run 326479, Event 1328354, LumiSection 7 on stream 1 at 30-Jun-2023 08:33:13.279 CEST
%MSG-w SiStripRawToDigi:  SiStripRawToDigiModule:siStripDigisHLT  30-Jun-2023 08:33:13 CEST Run: 326479 Event: 1402087
NULL pointer to FEDRawData for FED: id 434
Note: further warnings of this type will be suppressed (this can be changed by enabling debugging printout)
%MSG
%MSG-w SiStripRawToDigi:  SiStripRawToDigiModule:siStripDigisHLT  30-Jun-2023 08:33:13 CEST Run: 326479 Event: 1579493
NULL pointer to FEDRawData for FED: id 434
Note: further warnings of this type will be suppressed (this can be changed by enabling debugging printout)
%MSG
%MSG-w SiStripRawToDigi:  SiStripRawToDigiModule:siStripDigisHLT  30-Jun-2023 08:33:13 CEST Run: 326479 Event: 1328354
NULL pointer to FEDRawData for FED: id 434
Note: further warnings of this type will be suppressed (this can be changed by enabling debugging printout)
%MSG
%MSG-w SiStripRawToDigi:  SiStripRawToDigiModule:siStripDigisHLT  30-Jun-2023 08:33:13 CEST Run: 326479 Event: 1394020
NULL pointer to FEDRawData for FED: id 434
Note: further warnings of this type will be suppressed (this can be changed by enabling debugging printout)
%MSG
----- Begin Fatal Exception 30-Jun-2023 08:33:33 CEST-----------------------
An exception of category 'CorruptedData' occurred while
   [0] Processing  Event run: 326479 lumi: 7 event: 1328354 stream: 1
   [1] Running path 'REPACKRAWoutput_step'
   [2] Prefetching for module PoolOutputModule/'REPACKRAWoutput'
   [3] Calling method for module SiStripClusters2ApproxClusters/'hltSiStripClusters2ApproxClusters'
Exception Message:
[SiStripNoises::getNoise] looking for SiStripNoises for a strip out of range: strip 768
----- End Fatal Exception -------------------------------------------------
----- Begin Fatal Exception 30-Jun-2023 08:33:33 CEST-----------------------
An exception of category 'CorruptedData' occurred while
   [0] Processing  Event run: 326479 lumi: 7 event: 1402087 stream: 0
   [1] Running path 'REPACKRAWoutput_step'
   [2] Prefetching for module PoolOutputModule/'REPACKRAWoutput'
   [3] Calling method for module SiStripClusters2ApproxClusters/'hltSiStripClusters2ApproxClusters'
Exception Message:
[SiStripNoises::getNoise] looking for SiStripNoises for a strip out of range: strip 768
----- End Fatal Exception -------------------------------------------------
----- Begin Fatal Exception 30-Jun-2023 08:33:33 CEST-----------------------
An exception of category 'CorruptedData' occurred while
   [0] Processing  Event run: 326479 lumi: 7 event: 1579493 stream: 2
   [1] Running path 'REPACKRAWoutput_step'
   [2] Prefetching for module PoolOutputModule/'REPACKRAWoutput'
   [3] Calling method for module SiStripClusters2ApproxClusters/'hltSiStripClusters2ApproxClusters'
Exception Message:
[SiStripNoises::getNoise] looking for SiStripNoises for a strip out of range: strip 768
----- End Fatal Exception -------------------------------------------------
----- Begin Fatal Exception 30-Jun-2023 08:33:33 CEST-----------------------
An exception of category 'CorruptedData' occurred while
   [0] Processing  Event run: 326479 lumi: 7 event: 1394020 stream: 3
   [1] Running path 'REPACKRAWoutput_step'
   [2] Prefetching for module PoolOutputModule/'REPACKRAWoutput'
   [3] Calling method for module SiStripClusters2ApproxClusters/'hltSiStripClusters2ApproxClusters'
Exception Message:
[SiStripNoises::getNoise] looking for SiStripNoises for a strip out of range: strip 768

There are also warnings about NULL pointers to FEDRawData, not sure if they are related or harmless:

%MSG-w SiStripRawToDigi:  SiStripRawToDigiModule:siStripDigisHLT  30-Jun-2023 08:33:13 CEST Run: 326479 Event: 1402087
NULL pointer to FEDRawData for FED: id 434
Note: further warnings of this type will be suppressed (this can be changed by enabling debugging printout)
(...)
%MSG-w SiStripRawToDigi:  SiStripRawToDigiModule:siStripDigisHLT@endStream  30-Jun-2023 08:33:33 CEST PostEndProcessBlock
[sistrip::RawToDigiUnpacker::createDigis] warnings:
NULL pointer to FEDRawData for FED (1)

The text was updated successfully, but these errors were encountered:

iarspider · 2023-06-30T08:55:03Z

assign reconstruction

cmsbuild · 2023-06-30T08:55:16Z

New categories assigned: reconstruction

@mandrenguyen,@clacaputo you have been requested to review this Pull request/Issue and eventually sign? Thanks

cmsbuild · 2023-06-30T08:55:18Z

A new Issue was created by @iarspider .

@Dr15Jones, @perrotta, @dpiparo, @rappoccio, @makortel, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

mmusich · 2023-06-30T15:22:57Z

Both this and the other companion issue #42131 are caused because there is a condition (SiStripNoises) which is read off its domain from this call:

cmssw/RecoTracker/PixelLowPtUtilities/interface/SlidingPeakFinder.h

Line 72 in fe01c0e

    
           cut_ = std::min<float>(seedCutMIPs * mip, seedCutSN * noiseObj_->getNoise(firstStrip + 1, noises_));

when it looks for the noise of strip n. 768 for Detid 369120277.
I suppose this was triggered by #41815 (tagging also @Ksavva1021) .
I am not sure why the payload stored in the tag SiStripNoise_v2_prompt for run 326417 (which is the run used for wf 140.58) is missing this particular strip.

aandvalenzuela · 2023-07-18T13:18:56Z

Hello! There are three new workflows failing with the same issue for CMSSW_13_2 2023-07-13-2300 (latest DBG IB). I link the logs here: 159.03, 160.02 and 160.03.

Ksavva1021 · 2023-07-18T21:48:42Z

There is a workaround to this (to just not use that strip) but, still, the question stands as to why the payload stored in the tag SiStripNoise_v2_prompt for run 326417 (which is the run used for wf 140.58) is missing this particular strip as @mmusich mentioned.

makortel · 2023-07-31T14:10:16Z

assign alca

FYI @cms-sw/trk-dpg-l2

cmsbuild · 2023-07-31T14:10:37Z

New categories assigned: alca

@perrotta,@consuegs,@francescobrivio,@saumyaphor4252,@tvami you have been requested to review this Pull request/Issue and eventually sign? Thanks

tvami · 2023-08-01T22:28:33Z

I dont know how one can figure out why that strips is missing since this is a story from 2018 (Nov 9, 2018)... By putting "326417" into my gmail, I find that the payload is coming from an O2O from some HI running. The msg from Raffaele says "after the first HI fills of yesterday, we saw, as expected from the online calibrations loaded one week ago, a reduction in the S/N particularly visible in TID and TECs. With this o2o, a new set of pedestal and noise values measured yesterday are deployed offline. We can profit of the next cosmics/collisions to quickly validate them. In case they will create troubles, a roll back is possible in 10 minutes. As far as i can tell, no large differences observed in pedestals, reduction of noise in TEC/TID and inner TIB layers of about 5%." which is followed by an unanswered question from Andrea Venturi saying " I am wondering if there is a way to produce S/N distributions selecting only (on-track) clusters in APVs that were zero-suppressed in the standard way. Just to be sure that the hybrid ZS and the clusterizer do not introduce a reduction of S and, therefore, of S/N."

--> so was that strip ZS differently than the rest and somehow didnt enter the payload? That's all I can offer.

Actually here is the tracker map of the noise in the relevant payload: https://cern.ch/u3ouy

And I also pingged the strips convenors

mdelcourt · 2023-08-03T09:15:10Z

I checked a recent noise payload, and it seems that this strip never exists. For this detId (and similar modules), the strip range is from 0 to 767.

mmusich · 2023-08-03T09:24:04Z

One would presume that protecting this call:

cmssw/RecoLocalTracker/SiStripClusterizer/plugins/SiStripClusters2ApproxClusters.cc

Line 134 in c2e865a

PeakFinderTest test(mipnorm,

with the output of this filter:

cmssw/RecoLocalTracker/SiStripClusterizer/plugins/SiStripClusters2ApproxClusters.cc

Line 129 in c2e865a

theFilter->getSizes(detId, cluster, lp, ldir, hitStrips, hitPredPos);

would prevent reading off domain, but apparently this condition:

cmssw/RecoTracker/PixelLowPtUtilities/src/ClusterShapeHitFilter.cc

Line 349 in 2c98515

bool usable = (fs >= 1) & (fs + meas <= ns + 1);

is not restrictive enough.

makortel · 2023-08-03T13:49:58Z

Given that the access in

cmssw/RecoTracker/PixelLowPtUtilities/interface/SlidingPeakFinder.h

Line 72 in fe01c0e

    
           cut_ = std::min<float>(seedCutMIPs * mip, seedCutSN * noiseObj_->getNoise(firstStrip + 1, noises_));

is with firstStrip + 1 == 768, and firstStrip == 767 seems to be a valid value for a strip (i.e. assuming on the caller side

cmssw/RecoLocalTracker/SiStripClusterizer/plugins/SiStripClusters2ApproxClusters.cc

Line 134 in c2e865a

PeakFinderTest test(mipnorm,

observing the last strip of a module is ok), could it be the PeakFinderTest does not properly handle the edge case?

Actually, why does the PeakFinderTest inspect the noise of the strip next to the first strip of the cluster? (as opposed to e.g. the first strip, or "center strip" of the cluster)

mmusich · 2023-08-03T13:58:25Z

could it be the PeakFinderTest does not properly handle the edge case?

I think that's possiblity, but I am wondering why we observe this now (a very similar code was already used in

cmssw/RecoTracker/PixelLowPtUtilities/src/StripSubClusterShapeTrajectoryFilter.cc

Line 84 in c2e865a

struct PeakFinderTest {

) which I think it has been used since ever for the Strip seeded iterations.

Actually, why does the PeakFinderTest inspect the noise of the strip next to the first strip of the cluster? (as opposed to e.g. the first strip, or "center strip" of the cluster)

that's also the question I asked myself, but could not find an answer. I suspect all of this code is from Run1.

slava77 · 2023-08-03T16:00:05Z

Do I understand correctly that this is present only in DBG_X ? (not in other builds)

mmusich · 2023-08-03T16:08:07Z

Do I understand correctly that this is present only in DBG_X ? (not in other builds)

A similar issue happens in ASAN_X: #42131

makortel · 2023-08-03T18:58:35Z

Do I understand correctly that this is present only in DBG_X ? (not in other builds)

The exception occurs only in DBG_X builds because the verify() function (that does the check and throws the exception) is called only when EDM_ML_DEBUG is defined

cmssw/CondFormats/SiStripObjects/interface/SiStripNoises.h

Lines 73 to 78 in c2e865a

    
             static float getNoise(uint16_t strip, const Range& range) { 
        
           #ifdef EDM_ML_DEBUG 
        
               verify(strip, range); 
        
           #endif 
        
               return getNoiseFast(strip, range); 
        
             }

To me it looks like the issue is there in all builds, but doesn't show any (technical) sign exception in DBG_X and ASAN_X.

mmusich · 2023-08-07T08:26:07Z

type trk

mmusich · 2023-08-07T12:39:29Z

but I am wondering why we observe this now (a very similar code was already used in cmssw/RecoTracker/PixelLowPtUtilities/src/StripSubClusterShapeTrajectoryFilter.cc which I think it has been used since ever for the Strip seeded iterations.

answering to myself, this doesn't happen because of this check:

cmssw/RecoTracker/PixelLowPtUtilities/src/StripSubClusterShapeTrajectoryFilter.cc

Lines 265 to 267 in c2e865a

    
           if (std::abs(hitPredPos) < 1.5f && hitStrips <= 2) { 
        
             return true; 
        
           }

I propose the same kind of protection at #42486

tvami · 2023-08-09T01:01:34Z

+alca

resolved in SiStripClusters2ApproxClusters: run peakFilter only if cluster is usable #42486

cmsbuild added reconstruction-pending pending-signatures labels Jun 30, 2023

mmusich mentioned this issue Jun 30, 2023

[ASAN] heap-buffer-overflow in SiStripClusters2ApproxClusters::produce #42131

Closed

cmsbuild added the alca-pending label Jul 31, 2023

mmusich mentioned this issue Aug 1, 2023

[DBG] Fix comparison of integer expressions of different signedness #42405

Merged

cmsbuild added the trk label Aug 7, 2023

mmusich mentioned this issue Aug 7, 2023

SiStripClusters2ApproxClusters: run peakFilter only if cluster is usable #42486

Merged

cmsbuild closed this as completed in #42486 Aug 8, 2023

cmsbuild added alca-approved and removed alca-pending labels Aug 9, 2023

mmusich mentioned this issue Aug 9, 2023

[13.2.X] SiStripClusters2ApproxClusters: run peakFilter only if cluster is usable #42516

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DBG_X] RelVal 140.58 Step 2: Corrupted data in SiStripNoises::getNoise #42162

[DBG_X] RelVal 140.58 Step 2: Corrupted data in SiStripNoises::getNoise #42162

iarspider commented Jun 30, 2023

iarspider commented Jun 30, 2023

cmsbuild commented Jun 30, 2023

cmsbuild commented Jun 30, 2023

mmusich commented Jun 30, 2023 •

edited

Loading

aandvalenzuela commented Jul 18, 2023

Ksavva1021 commented Jul 18, 2023 •

edited

Loading

makortel commented Jul 31, 2023

cmsbuild commented Jul 31, 2023

tvami commented Aug 1, 2023 •

edited

Loading

mdelcourt commented Aug 3, 2023

mmusich commented Aug 3, 2023

makortel commented Aug 3, 2023

mmusich commented Aug 3, 2023 •

edited

Loading

slava77 commented Aug 3, 2023

mmusich commented Aug 3, 2023

makortel commented Aug 3, 2023

mmusich commented Aug 7, 2023

mmusich commented Aug 7, 2023

tvami commented Aug 9, 2023

[DBG_X] RelVal 140.58 Step 2: Corrupted data in SiStripNoises::getNoise #42162

[DBG_X] RelVal 140.58 Step 2: Corrupted data in SiStripNoises::getNoise #42162

Comments

iarspider commented Jun 30, 2023

iarspider commented Jun 30, 2023

cmsbuild commented Jun 30, 2023

cmsbuild commented Jun 30, 2023

mmusich commented Jun 30, 2023 • edited Loading

aandvalenzuela commented Jul 18, 2023

Ksavva1021 commented Jul 18, 2023 • edited Loading

makortel commented Jul 31, 2023

cmsbuild commented Jul 31, 2023

tvami commented Aug 1, 2023 • edited Loading

mdelcourt commented Aug 3, 2023

mmusich commented Aug 3, 2023

makortel commented Aug 3, 2023

mmusich commented Aug 3, 2023 • edited Loading

slava77 commented Aug 3, 2023

mmusich commented Aug 3, 2023

makortel commented Aug 3, 2023

mmusich commented Aug 7, 2023

mmusich commented Aug 7, 2023

tvami commented Aug 9, 2023

mmusich commented Jun 30, 2023 •

edited

Loading

Ksavva1021 commented Jul 18, 2023 •

edited

Loading

tvami commented Aug 1, 2023 •

edited

Loading

mmusich commented Aug 3, 2023 •

edited

Loading