large number of tracking-related warning/error messages in HLT jobs #39234

missirol · 2022-08-29T13:22:12Z

CMSSW jobs with recent HLT menus [0] show a very large number of LogWarnings and LogErrors coming mainly from the HLT e/gamma [1] and muon [2] reconstruction. The warnings/errors seem to be mainly related to tracking. A recipe to reproduce some of these warnings/errors is in [3].

In CMSHLT-2449, it was noted that this creates significant size increase of HLT log files, and the short-term 'solution' was to silence these warnings (not the errors) via the MessageLogger module of the HLT menu.

This issue is about understanding from RECO and POG experts what the best solution is.

Are these messages a sign of issues that should be addressed with some urgency?
If not, should they be downgraded to "info" or "debug" messages, rather than being silenced by HLT?

FYI: @fwyzard @Martin-Grunewald @silviodonato @swagata87 @JanFSchulte

[0] I currently don't know when this started, and if it is different with earlier releases or menus.

[1]

%MSG-w BasicTrajectoryState:  GsfTrackProducer:hltEgammaGsfTracksForBParking  29-Aug-2022 14:51:04 CEST Run: 355558 Event: 215872193
curv error not pos-def
[      -663558      29.038      483458   1.241e+07     -314550
        29.038   -0.245501    -20.6987      -535.1     6.49207
        483458    -20.6987     -352241-9.04176e+06      229186
     1.241e+07      -535.1-9.04176e+06-2.32095e+08   5.883e+06
       -314550     6.49207      229186   5.883e+06     -149378 ]
pos/mom/mf  (30.1516,49.9366,-35.9222)   (2.33339,3.34254,-2.93736)   (-0.00243308,-0.00402963,3.81145)
%MSG
%MSG-e PosteriorWeightsCalculator:   GsfTrackProducer:hltEgammaGsfTracksForBParking 29-Aug-2022 14:51:04 CEST  Run: 355558 Event: 215872193
PosteriorWeightsCalculator: sumWeight < DBL_MIN
%MSG
%MSG-e GsfMultiStateUpdator:  GsfTrackProducer:hltEgammaGsfTracksForBParking  29-Aug-2022 14:51:04 CEST Run: 355558 Event: 215872193
 no weights could be retreived. invalid updated state !.
%MSG
%MSG-e InvalidState:  GsfTrackProducer:hltEgammaGsfTracksForBParking  29-Aug-2022 14:51:04 CEST Run: 355558 Event: 215872193
inside hit

[2]

%MSG-w RecoMuon:  L3MuonProducer:hltL3NoFiltersNoVtxMuonsOIHit  29-Aug-2022 14:50:53 CEST Run: 355558 Event: 216016013                                                                             
     Failed to load tracker track trajectory

[3]

#!/bin/bash

cmsrel CMSSW_12_4_8
cd CMSSW_12_4_8/src
cmsenv

hltGetConfiguration /online/collisions/2022/2e34/v1.3/HLT/V1 \
  --globaltag 124X_dataRun3_HLT_v4 \
  --data \
  --unprescale \
  --output none \
  --max-events 200 \
  --eras Run3 --l1-emulator uGT --l1 L1Menu_Collisions2022_v1_3_0-d1_xml \
  --input root://eoscms.cern.ch//eos/cms/store/group/dpg_trigger/comm_trigger/TriggerStudiesGroup/STORM/RAW/Run2022B_HLTPhysics0_run355558/cd851cf4-0fca-4d76-b80e-1d33e1371929.root \
  > hlt.py

cmsRun hlt.py &> hlt.log

The text was updated successfully, but these errors were encountered:

cmsbuild · 2022-08-29T13:22:35Z

A new Issue was created by @missirol Marino Missiroli.

@Dr15Jones, @perrotta, @dpiparo, @rappoccio, @makortel, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

makortel · 2022-08-29T13:34:15Z

assign reconstruction

cmsbuild · 2022-08-29T13:34:34Z

New categories assigned: reconstruction

@jpata,@clacaputo,@mandrenguyen you have been requested to review this Pull request/Issue and eventually sign? Thanks

makortel · 2022-08-29T13:34:40Z

FYI @cms-sw/egamma-pog-l2 @cms-sw/muon-pog-l2 @cms-sw/tracking-pog-l2

mmusich · 2022-08-29T14:35:34Z

error in [1] is the same thing I've already reported in #39026 (comment).

swagata87 · 2022-08-29T14:42:43Z

about this message:

%MSG-w BasicTrajectoryState:  GsfTrackProducer:hltEgammaGsfTracksForBParking  29-Aug-2022 14:51:04 CEST Run: 355558 Event: 215872193
curv error not pos-def
[      -663558      29.038      483458   1.241e+07     -314550
        29.038   -0.245501    -20.6987      -535.1     6.49207
        483458    -20.6987     -352241-9.04176e+06      229186
     1.241e+07      -535.1-9.04176e+06-2.32095e+08   5.883e+06
       -314550     6.49207      229186   5.883e+06     -149378 ]
pos/mom/mf  (30.1516,49.9366,-35.9222)   (2.33339,3.34254,-2.93736)   (-0.00243308,-0.00402963,3.81145)
%MSG

and similar common messages, raised from

cmssw/TrackingTools/TrajectoryState/src/BasicTrajectoryState.cc

Lines 87 to 97 in 2a2ca2e

    
           void verifyLocalErr(LocalTrajectoryError const& err, const FreeTrajectoryState& state) { 
        
             if UNLIKELY (!err.posDef()) 
        
               edm::LogWarning("BasicTrajectoryState") << "local error not pos-def\n" 
        
                                                       << err.matrix() << "\npos/mom/mf " << state.position() << ' ' 
        
                                                       << state.momentum() << ' ' << state.parameters().magneticFieldInTesla(); 
        
           } 
        
           void verifyCurvErr(CurvilinearTrajectoryError const& err, const FreeTrajectoryState& state) { 
        
             if UNLIKELY (!err.posDef()) 
        
               edm::LogWarning("BasicTrajectoryState") << "curv error not pos-def\n" 
        
                                                       << err.matrix() << "\npos/mom/mf " << state.position() << ' ' 
        
                                                       << state.momentum() << ' ' << state.parameters().magneticFieldInTesla();

, I have a feeling that the warning is printing too much information. We can just print .... error not pos-def , and remove the details like err.matrix(), state.position(), state.momentum() , state.parameters().magneticFieldInTesla().
Will that be acceptable?
(let me also tag @VinInn in case he has suggestions about this topic)

fwyzard · 2022-08-29T14:45:36Z

Will that be acceptable?

No.

If the warning is something that needs to be followed up, then people should look into it within one-two weeks at most, and solve the underlying issue.

If there is nothing to fix, i.e. code is behaving as expected and the physics results are correct, then the warning should be downgraded to LogInfo or even better LogDebug.

slava77 · 2022-08-29T15:27:14Z

Will that be acceptable?

No.

If the warning is something that needs to be followed up, then people should look into it within one-two weeks at most, and solve the underlying issue.

If there is nothing to fix, i.e. code is behaving as expected and the physics results are correct, then the warning should be downgraded to LogInfo or even better LogDebug.

That's more of a wishful thinking, not how warnings were treated in the last 10+ years.

fwyzard · 2022-08-29T15:50:27Z

I cannot change how RECO and others handled things in the past. I think I _can_ improve how things are being handled now. For example, let's pull from the online and offline reconstruction the offending modules, and see how long it takes to remove the warnings ?

swagata87 · 2022-08-31T03:20:59Z

It might be useful to keep the LogWarning messages in TrackingTools/TrajectoryState/src/BasicTrajectoryState.cc, as they are helpful to discover other bugs while code-development, debugging crashes that shows up, etc. With this PR (#39256), the warnings/errors are significantly reduced at HLT.

fwyzard · 2022-08-31T05:25:12Z

If I read the code correctly, #39256 is making the check more stringent, from

if (updatedTSOS.isValid() && updatedTSOS.localError().valid()) {

to

if (updatedTSOS.isValid() && updatedTSOS.localError().valid() && updatedTSOS.localError().posDef() && updatedTSOS.curvilinearError().matrix().Det2(det) && det > 0) {

Since the LogError is in the "else" branch:

if (...) {
   result.addState(...);
} else {
   edm::LogError("GsfMultiStateUpdator") << "KF updated state " << i << " is invalid. skipping.";
}

the change should increase the amount of errors being reported, not reduce them.

fwyzard · 2022-08-31T05:25:45Z

It might be useful to keep the LogWarning messages in TrackingTools/TrajectoryState/src/BasicTrajectoryState.cc, as they are helpful to discover other bugs while code-development, debugging crashes that shows up, etc.

That is what LogDebug is for.

mmusich · 2022-08-31T05:50:04Z

the change should increase the amount of errors being reported, not reduce them.

case in point no, because the LogError that plagues the HLT logs is not the one originating from taking the else branch of that decision, but from taking the passing one, leading to trying to combine inconsistent states in the multistate. #39026 (comment) offers some analysis of the call stack.

fwyzard · 2022-08-31T06:50:59Z

I see, so we get a reduction in %MSG-w BasicTrajectoryState without increasing the number of errors ?

-%MSG-w BasicTrajectoryState:  GsfTrackProducer:hltEgammaGsfTracksForBParking  31-Aug-2022 08:18:49 CEST Run: 357329 Event: 65599505
-curv error not pos-def
-[   3.7613e+07     294.713-2.89415e+06    -6909.75    -2081.74
-       294.713  0.00433287    -22.6764  -0.0489485   0.0109365
-  -2.89415e+06    -22.6764      222691     531.393     160.185
-      -6909.75  -0.0489485     531.393    -2.46602    0.454109
-      -2081.74   0.0109365     160.185    0.454109    0.482128 ]
-pos/mom/mf  (7.8834,-14.3362,-6.56777)   (10.0654,-18.2823,2.70992)   (-0.000146921,0.00026718,3.81131)
-%MSG
-%MSG-w BasicTrajectoryState:  GsfTrackProducer:hltEgammaGsfTracksForBParking  31-Aug-2022 08:18:49 CEST Run: 357329 Event: 65599505
-local error not pos-def
-[   3.7613e+07 2.89715e+06      -15369     6915.44     2063.28
-   2.89715e+06      223154     -1183.8     532.384     158.932
-        -15369     -1183.8     6.28202    -2.81894   -0.815183
-       6915.44     532.384    -2.81894    -2.46759    0.471503
-       2063.28     158.932   -0.815183    0.471503    0.485435 ]
-pos/mom/mf  (7.8834,-14.3362,-6.56777)   (10.0654,-18.2823,2.70992)   (-0.000146921,0.00026718,3.81131)
-%MSG
-%MSG-e PosteriorWeightsCalculator:   GsfTrackProducer:hltEgammaGsfTracksForBParking 31-Aug-2022 08:18:49 CEST  Run: 357329 Event: 65599505
-PosteriorWeightsCalculator: sumWeight < DBL_MIN
-%MSG
-%MSG-e GsfMultiStateUpdator:  GsfTrackProducer:hltEgammaGsfTracksForBParking  31-Aug-2022 08:18:49 CEST Run: 357329 Event: 65599505
- no weights could be retreived. invalid updated state !.
-%MSG
-%MSG-e InvalidState:  GsfTrackProducer:hltEgammaGsfTracksForBParking  31-Aug-2022 08:18:49 CEST Run: 357329 Event: 65599505
-inside hit
-%MSG
+%MSG-e GsfMultiStateUpdator:  GsfTrackProducer:hltEgammaGsfTracksForBParking  31-Aug-2022 08:29:18 CEST Run: 357329 Event: 65599505
+KF updated state 0 is invalid. skipping.
+%MSG
+%MSG-e InvalidState:  GsfTrackProducer:hltEgammaGsfTracksForBParking  31-Aug-2022 08:29:18 CEST Run: 357329 Event: 65599505
+first hit
+%MSG
-%MSG-w BasicTrajectoryState:  GsfTrackProducer:hltEgammaGsfTracksForBParking  31-Aug-2022 08:18:49 CEST Run: 357329 Event: 65599505
-curv error not pos-def
-[   3.7613e+07     294.713-2.89415e+06    -6909.75    -2081.74
-       294.713  0.00433287    -22.6764  -0.0489485   0.0109365
-  -2.89415e+06    -22.6764      222691     531.393     160.185
-      -6909.75  -0.0489485     531.393    -2.46602    0.454109
-      -2081.74   0.0109365     160.185    0.454109    0.482128 ]
-pos/mom/mf  (7.8834,-14.3362,-6.56777)   (10.0654,-18.2823,2.70992)   (-0.000146921,0.00026718,3.81131)
-%MSG
-%MSG-w BasicTrajectoryState:  GsfTrackProducer:hltEgammaGsfTracksForBParking  31-Aug-2022 08:18:49 CEST Run: 357329 Event: 65599505
-local error not pos-def
-[   3.7613e+07 2.89715e+06      -15369     6915.44     2063.28
-   2.89715e+06      223154     -1183.8     532.384     158.932
-        -15369     -1183.8     6.28202    -2.81894   -0.815183
-       6915.44     532.384    -2.81894    -2.46759    0.471503
-       2063.28     158.932   -0.815183    0.471503    0.485435 ]
-pos/mom/mf  (7.8834,-14.3362,-6.56777)   (10.0654,-18.2823,2.70992)   (-0.000146921,0.00026718,3.81131)
-%MSG
-%MSG-e PosteriorWeightsCalculator:   GsfTrackProducer:hltEgammaGsfTracksForBParking 31-Aug-2022 08:18:49 CEST  Run: 357329 Event: 65599505
-PosteriorWeightsCalculator: sumWeight < DBL_MIN
-%MSG
-%MSG-e GsfMultiStateUpdator:  GsfTrackProducer:hltEgammaGsfTracksForBParking  31-Aug-2022 08:18:49 CEST Run: 357329 Event: 65599505
- no weights could be retreived. invalid updated state !.
-%MSG
-%MSG-e InvalidState:  GsfTrackProducer:hltEgammaGsfTracksForBParking  31-Aug-2022 08:18:49 CEST Run: 357329 Event: 65599505
-inside hit
-%MSG
+%MSG-e GsfMultiStateUpdator:  GsfTrackProducer:hltEgammaGsfTracksForBParking  31-Aug-2022 08:29:18 CEST Run: 357329 Event: 65599505
+KF updated state 0 is invalid. skipping.
+%MSG
+%MSG-e InvalidState:  GsfTrackProducer:hltEgammaGsfTracksForBParking  31-Aug-2022 08:29:18 CEST Run: 357329 Event: 65599505
+first hit
+%MSG

-%MSG-w BasicTrajectoryState:  GsfTrackProducer:hltEgammaGsfTracks  31-Aug-2022 08:18:49 CEST Run: 357329 Event: 65033984
-curv error not pos-def
-[  4.50868e+07    -492.822-1.58079e+06    -168.574    -232.861
-      -492.822  0.00583096     17.2786 0.000881737  0.00526915
-  -1.58079e+06     17.2786       55424     5.89368     8.16263
-      -168.574 0.000881737     5.89368  -0.0576909  -0.0051147
-      -232.861  0.00526915     8.16263  -0.0051147   0.0179101 ]
-pos/mom/mf  (5.49633,-3.94086,-6.28259)   (8.86537,-6.21352,-15.619)   (-9.90109e-05,7.09906e-05,3.81112)
-%MSG
-%MSG-w BasicTrajectoryState:  GsfTrackProducer:hltEgammaGsfTracks  31-Aug-2022 08:18:49 CEST Run: 357329 Event: 65033984
-local error not pos-def
-[  4.50868e+07 1.58727e+06     -160089    -168.434      425.45
-   1.58727e+06     55879.4    -5635.89     -5.9129     14.9731
-       -160089    -5635.89      568.43     0.59339    -1.52462
-      -168.434     -5.9129     0.59339   -0.057925   0.0148514
-        425.45     14.9731    -1.52462   0.0148514   0.0527617 ]
-pos/mom/mf  (5.49633,-3.94086,-6.28259)   (8.86537,-6.21352,-15.619)   (-9.90109e-05,7.09906e-05,3.81112)
-%MSG
-%MSG-e PosteriorWeightsCalculator:  GsfTrackProducer:hltEgammaGsfTracks  31-Aug-2022 08:18:49 CEST Run: 357329 Event: 65033984
-PosteriorWeightsCalculator: sumWeight < DBL_MIN
-%MSG
-%MSG-e GsfMultiStateUpdator:  GsfTrackProducer:hltEgammaGsfTracks  31-Aug-2022 08:18:49 CEST Run: 357329 Event: 65033984
- no weights could be retreived. invalid updated state !.
-%MSG
-%MSG-e InvalidState:  GsfTrackProducer:hltEgammaGsfTracks  31-Aug-2022 08:18:49 CEST Run: 357329 Event: 65033984
-inside hit
-%MSG
+%MSG-e GsfMultiStateUpdator:  GsfTrackProducer:hltEgammaGsfTracks  31-Aug-2022 08:29:19 CEST Run: 357329 Event: 65033984
+KF updated state 0 is invalid. skipping.
+%MSG
+%MSG-e GsfMultiStateUpdator:  GsfTrackProducer:hltEgammaGsfTracks  31-Aug-2022 08:29:19 CEST Run: 357329 Event: 65033984
+KF updated state 1 is invalid. skipping.  
+%MSG
+%MSG-e GsfMultiStateUpdator:  GsfTrackProducer:hltEgammaGsfTracks  31-Aug-2022 08:29:19 CEST Run: 357329 Event: 65033984
+KF updated state 2 is invalid. skipping. 
+%MSG    
+%MSG-e GsfMultiStateUpdator:  GsfTrackProducer:hltEgammaGsfTracks  31-Aug-2022 08:29:19 CEST Run: 357329 Event: 65033984
+KF updated state 3 is invalid. skipping.
+%MSG
+%MSG-e GsfMultiStateUpdator:  GsfTrackProducer:hltEgammaGsfTracks  31-Aug-2022 08:29:19 CEST Run: 357329 Event: 65033984
+KF updated state 5 is invalid. skipping.
+%MSG
+%MSG-e GsfMultiStateUpdator:  GsfTrackProducer:hltEgammaGsfTracks  31-Aug-2022 08:29:19 CEST Run: 357329 Event: 65033984
+KF updated state 11 is invalid. skipping.
+%MSG
+%MSG-e GsfMultiStateUpdator:  GsfTrackProducer:hltEgammaGsfTracks  31-Aug-2022 08:29:19 CEST Run: 357329 Event: 65033984
+KF updated state 0 is invalid. skipping.
+%MSG
+%MSG-e InvalidState:  GsfTrackProducer:hltEgammaGsfTracks  31-Aug-2022 08:29:19 CEST Run: 357329 Event: 65033984
+first hit
+%MSG

fwyzard · 2022-08-31T06:55:52Z

Running over 10k events, we go from 4883 warnings and errors

      2 %MSG-e CTPPSPixelDataFormatter:  CTPPSPixelRawToDigi:hltCTPPSPixelDigis
     33 %MSG-e GsfMultiStateUpdator:  GsfTrackProducer:hltEgammaGsfTracks
     16 %MSG-e GsfMultiStateUpdator:  GsfTrackProducer:hltEgammaGsfTracksForBParking
     33 %MSG-e InvalidState:  GsfTrackProducer:hltEgammaGsfTracks
     16 %MSG-e InvalidState:  GsfTrackProducer:hltEgammaGsfTracksForBParking
     16 %MSG-e PosteriorWeightsCalculator:   GsfTrackProducer:hltEgammaGsfTracksForBParking
     33 %MSG-e PosteriorWeightsCalculator:  GsfTrackProducer:hltEgammaGsfTracks
      1 %MSG-e TrajectoryNotPosDef:   TrackProducer:hltIterL3OIGlbDisplacedMuonCtfWithMaterialTracks
      1 %MSG-e TrajectoryNotPosDef:   TrackProducer:hltIterL3OIMuCtfWithMaterialTracksNoVtx
     75 %MSG-e TrajectoryNotPosDef:   TrackProducer:hltL3NoFiltersTkTracksFromL2IOHitNoVtx
      1 %MSG-e TrajectoryNotPosDef:  TrackProducer:hltL3TkTracksFromL2OIStateNoVtx
    178 %MSG-w BasicTrajectoryState:   CkfTrackCandidateMaker:hltIterL3OITrackCandidatesNoVtx
     34 %MSG-w BasicTrajectoryState:   TrackProducer:hltL3NoFiltersTkTracksFromL2IOHitNoVtx
     54 %MSG-w BasicTrajectoryState:  CkfTrackCandidateMaker:hltIterL3OITrackCandidates
   2232 %MSG-w BasicTrajectoryState:  GsfTrackProducer:hltEgammaGsfTracks
    186 %MSG-w BasicTrajectoryState:  GsfTrackProducer:hltEgammaGsfTracksForBParking
     11 %MSG-w GlobalMuon:  L3MuonProducer:hltIterL3GlbMuon
      8 %MSG-w GlobalMuon:  L3MuonProducer:hltIterL3GlbdTksDisplacedMuons
      8 %MSG-w GlobalMuon:  L3MuonProducer:hltL3GlbDisplacedMuonsIterL3OI
     15 %MSG-w GlobalMuon:  L3MuonProducer:hltL3MuonsIterL3OI
    256 %MSG-w GlobalMuon:  L3MuonProducer:hltL3MuonsIterL3OINoVtx
    116 %MSG-w GlobalMuon:  L3MuonProducer:hltL3NoFiltersNoVtxMuonsOIHit
     73 %MSG-w GlobalMuon:  L3MuonProducer:hltL3NoFiltersNoVtxMuonsOIState
      8 %MSG-w GlobalTrajectoryBuilderBase:   L3MuonProducer:hltIterL3GlbdTksDisplacedMuons
      8 %MSG-w GlobalTrajectoryBuilderBase:   L3MuonProducer:hltL3GlbDisplacedMuonsIterL3OI
    116 %MSG-w GlobalTrajectoryBuilderBase:   L3MuonProducer:hltL3NoFiltersNoVtxMuonsOIHit
     73 %MSG-w GlobalTrajectoryBuilderBase:   L3MuonProducer:hltL3NoFiltersNoVtxMuonsOIState
     11 %MSG-w GlobalTrajectoryBuilderBase:  L3MuonProducer:hltIterL3GlbMuon
     15 %MSG-w GlobalTrajectoryBuilderBase:  L3MuonProducer:hltL3MuonsIterL3OI
    256 %MSG-w GlobalTrajectoryBuilderBase:  L3MuonProducer:hltL3MuonsIterL3OINoVtx
      1 %MSG-w InvalidInputTag:  CSCHaloDataProducer:hltCSCHaloData
     11 %MSG-w Muon:  L3MuonProducer:hltIterL3GlbMuon
      8 %MSG-w Muon:  L3MuonProducer:hltIterL3GlbdTksDisplacedMuons
      8 %MSG-w Muon:  L3MuonProducer:hltL3GlbDisplacedMuonsIterL3OI
     15 %MSG-w Muon:  L3MuonProducer:hltL3MuonsIterL3OI
    256 %MSG-w Muon:  L3MuonProducer:hltL3MuonsIterL3OINoVtx
    117 %MSG-w Muon:  L3MuonProducer:hltL3NoFiltersNoVtxMuonsOIHit
     74 %MSG-w Muon:  L3MuonProducer:hltL3NoFiltersNoVtxMuonsOIState
      1 %MSG-w MuonUpdatorAtVertex:  L3MuonProducer:hltL3NoFiltersNoVtxMuonsOIHit
      1 %MSG-w MuonUpdatorAtVertex:  L3MuonProducer:hltL3NoFiltersNoVtxMuonsOIState
     11 %MSG-w RecoMuon:  L3MuonProducer:hltIterL3GlbMuon
      8 %MSG-w RecoMuon:  L3MuonProducer:hltIterL3GlbdTksDisplacedMuons
      8 %MSG-w RecoMuon:  L3MuonProducer:hltL3GlbDisplacedMuonsIterL3OI
     15 %MSG-w RecoMuon:  L3MuonProducer:hltL3MuonsIterL3OI
    256 %MSG-w RecoMuon:  L3MuonProducer:hltL3MuonsIterL3OINoVtx
    117 %MSG-w RecoMuon:  L3MuonProducer:hltL3NoFiltersNoVtxMuonsOIHit
     74 %MSG-w RecoMuon:  L3MuonProducer:hltL3NoFiltersNoVtxMuonsOIState
     15 %MSG-w TrackNaN:  TrackProducer:hltL3NoFiltersTkTracksFromL2IOHitNoVtx
      2 %MSG-w TwoTrackMinimumDistance:   HLTDisplacedmumuVtxProducer:hltDisplacedmumuVtxNoMatchingProducer

to only 2424:

      3 %MSG-e CTPPSPixelDataFormatter:  CTPPSPixelRawToDigi:hltCTPPSPixelDigis
     40 %MSG-e GsfMultiStateUpdator:  GsfTrackProducer:hltEgammaGsfTracks
     16 %MSG-e GsfMultiStateUpdator:  GsfTrackProducer:hltEgammaGsfTracksForBParking
     33 %MSG-e InvalidState:  GsfTrackProducer:hltEgammaGsfTracks
     16 %MSG-e InvalidState:  GsfTrackProducer:hltEgammaGsfTracksForBParking
      1 %MSG-e TrajectoryNotPosDef:   TrackProducer:hltIterL3OIGlbDisplacedMuonCtfWithMaterialTracks
      1 %MSG-e TrajectoryNotPosDef:   TrackProducer:hltIterL3OIMuCtfWithMaterialTracksNoVtx
     75 %MSG-e TrajectoryNotPosDef:   TrackProducer:hltL3NoFiltersTkTracksFromL2IOHitNoVtx
      1 %MSG-e TrajectoryNotPosDef:  TrackProducer:hltL3TkTracksFromL2OIStateNoVtx
    178 %MSG-w BasicTrajectoryState:   CkfTrackCandidateMaker:hltIterL3OITrackCandidatesNoVtx
     34 %MSG-w BasicTrajectoryState:   TrackProducer:hltL3NoFiltersTkTracksFromL2IOHitNoVtx
     54 %MSG-w BasicTrajectoryState:  CkfTrackCandidateMaker:hltIterL3OITrackCandidates
     11 %MSG-w GlobalMuon:  L3MuonProducer:hltIterL3GlbMuon
      8 %MSG-w GlobalMuon:  L3MuonProducer:hltIterL3GlbdTksDisplacedMuons
      8 %MSG-w GlobalMuon:  L3MuonProducer:hltL3GlbDisplacedMuonsIterL3OI
     15 %MSG-w GlobalMuon:  L3MuonProducer:hltL3MuonsIterL3OI
    256 %MSG-w GlobalMuon:  L3MuonProducer:hltL3MuonsIterL3OINoVtx
    116 %MSG-w GlobalMuon:  L3MuonProducer:hltL3NoFiltersNoVtxMuonsOIHit
     73 %MSG-w GlobalMuon:  L3MuonProducer:hltL3NoFiltersNoVtxMuonsOIState
      8 %MSG-w GlobalTrajectoryBuilderBase:   L3MuonProducer:hltIterL3GlbdTksDisplacedMuons
      8 %MSG-w GlobalTrajectoryBuilderBase:   L3MuonProducer:hltL3GlbDisplacedMuonsIterL3OI
    116 %MSG-w GlobalTrajectoryBuilderBase:   L3MuonProducer:hltL3NoFiltersNoVtxMuonsOIHit
     73 %MSG-w GlobalTrajectoryBuilderBase:   L3MuonProducer:hltL3NoFiltersNoVtxMuonsOIState
     11 %MSG-w GlobalTrajectoryBuilderBase:  L3MuonProducer:hltIterL3GlbMuon
     15 %MSG-w GlobalTrajectoryBuilderBase:  L3MuonProducer:hltL3MuonsIterL3OI
    256 %MSG-w GlobalTrajectoryBuilderBase:  L3MuonProducer:hltL3MuonsIterL3OINoVtx
      1 %MSG-w InvalidInputTag:  CSCHaloDataProducer:hltCSCHaloData
     11 %MSG-w Muon:  L3MuonProducer:hltIterL3GlbMuon
      8 %MSG-w Muon:  L3MuonProducer:hltIterL3GlbdTksDisplacedMuons
      8 %MSG-w Muon:  L3MuonProducer:hltL3GlbDisplacedMuonsIterL3OI
     15 %MSG-w Muon:  L3MuonProducer:hltL3MuonsIterL3OI
    256 %MSG-w Muon:  L3MuonProducer:hltL3MuonsIterL3OINoVtx
    117 %MSG-w Muon:  L3MuonProducer:hltL3NoFiltersNoVtxMuonsOIHit
     74 %MSG-w Muon:  L3MuonProducer:hltL3NoFiltersNoVtxMuonsOIState
      1 %MSG-w MuonUpdatorAtVertex:  L3MuonProducer:hltL3NoFiltersNoVtxMuonsOIHit
      1 %MSG-w MuonUpdatorAtVertex:  L3MuonProducer:hltL3NoFiltersNoVtxMuonsOIState
     11 %MSG-w RecoMuon:  L3MuonProducer:hltIterL3GlbMuon
      8 %MSG-w RecoMuon:  L3MuonProducer:hltIterL3GlbdTksDisplacedMuons
      8 %MSG-w RecoMuon:  L3MuonProducer:hltL3GlbDisplacedMuonsIterL3OI
     15 %MSG-w RecoMuon:  L3MuonProducer:hltL3MuonsIterL3OI
    256 %MSG-w RecoMuon:  L3MuonProducer:hltL3MuonsIterL3OINoVtx
    117 %MSG-w RecoMuon:  L3MuonProducer:hltL3NoFiltersNoVtxMuonsOIHit
     74 %MSG-w RecoMuon:  L3MuonProducer:hltL3NoFiltersNoVtxMuonsOIState
     15 %MSG-w TrackNaN:  TrackProducer:hltL3NoFiltersTkTracksFromL2IOHitNoVtx
      2 %MSG-w TwoTrackMinimumDistance:   HLTDisplacedmumuVtxProducer:hltDisplacedmumuVtxNoMatchingProducer

the differences being a large reduction in warnings and only a small increase in errors:

-     33 %MSG-e GsfMultiStateUpdator:  GsfTrackProducer:hltEgammaGsfTracks
+     40 %MSG-e GsfMultiStateUpdator:  GsfTrackProducer:hltEgammaGsfTracks
-     16 %MSG-e PosteriorWeightsCalculator:   GsfTrackProducer:hltEgammaGsfTracksForBParking
-     33 %MSG-e PosteriorWeightsCalculator:  GsfTrackProducer:hltEgammaGsfTracks
-   2232 %MSG-w BasicTrajectoryState:  GsfTrackProducer:hltEgammaGsfTracks
-    186 %MSG-w BasicTrajectoryState:  GsfTrackProducer:hltEgammaGsfTracksForBParking

mmusich · 2022-08-31T06:56:32Z

I see, so we get a reduction in %MSG-w BasicTrajectoryState without increasing the number of errors ?

I think in a nutshell #39256 is exchanging a more verbose error for multiple less verbose ones. One might wonder why these pathological states even happen to exist (which is where I think the investigation should be moved to)

missirol · 2022-09-05T05:52:58Z

@swagata87 , after your PR(s) , do you think we can remove hltEgammaGsfTracks* from the suppressWarning list of the MessageLogger module in the HLT menu?

@JanFSchulte , what would you suggest to do about the many warnings like [2]?

[2]

%MSG-w RecoMuon:  L3MuonProducer:hltL3NoFiltersNoVtxMuonsOIHit  29-Aug-2022 14:50:53 CEST Run: 355558 Event: 216016013                                                                             
     Failed to load tracker track trajectory

swagata87 · 2022-09-05T08:25:55Z

Hello Marino,
yes, you can remove hltEgammaGsfTracks from the suppressWarning list.
We also plan to work on the remaining error messages, but that's a bit more long-term thing.
Thanks,
Swagata.

wonpoint4 · 2022-10-03T10:22:30Z

Dear expert,
We think these warning [2] doesn't seem really problematic in the muon reconstruction, so we can proceed to remove these warning in CMSSW. I think the LogWarning could be demoted to LogInfo or LogDebug.
Sincerely,
Won

missirol · 2022-10-03T17:17:44Z

I think the LogWarning could be demoted to LogInfo or LogDebug.

@wonpoint4 , okay, can you please prepare this PR so experts can review it? Thanks.

missirol · 2022-10-04T12:51:41Z

I think the LogWarning could be demoted to LogInfo or LogDebug.

The MUO POG has opened the corresponding PR in #39596.

missirol · 2022-10-14T17:00:20Z

I think the original issue has been addressed by EGM and MUO POGs, by tweaking or demoting the warning messages. At least now HLT is not forced anymore to mask the warnings of those producers.

It's not clear to me if there are still reconstruction issues to follow up on (see #39234 (comment)). I leave it to others to continue that discussion in this issue, or to do so in a separate one if needed.

mmusich · 2022-10-14T17:13:43Z

I leave it to others to continue that discussion in this issue, or to do so in a separate one if needed.

Trying to follow-up on a seemingly unrelated issue, I stumbled on the fact that in the alca FullTrack validation produced for one of the Tracker alignment updates, the number of e/gamma @ HLT related messages was drastically reduced when using the more updated conditions. Perhaps it's a lead to the actual problem.

cmsbuild added the pending-assignment label Aug 29, 2022

cmsbuild added pending-signatures reconstruction-pending and removed pending-assignment labels Aug 29, 2022

swagata87 mentioned this issue Aug 31, 2022

[GSF tracking] Add TSOS only if it is meaningful #39256

Merged

This was referenced Aug 31, 2022

[124X backport] [GSF tracking] Add TSOS only if it is meaningful #39275

Merged

[125X backport] [GSF tracking] Add TSOS only if it is meaningful #39276

Merged

khaosmos93 mentioned this issue Oct 4, 2022

[MUO] Suppress excessive warnings in muon tracking at HLT #39596

Merged

khaosmos93 mentioned this issue Oct 4, 2022

[Backport 124X] [MUO] Suppress excessive warnings in muon tracking at HLT #39601

Merged

khaosmos93 mentioned this issue Oct 4, 2022

[Backport 125X] [MUO] Suppress excessive warnings in muon tracking at HLT #39602

Merged

mmusich mentioned this issue Oct 24, 2022

GsfTracking crash at HLT in run 359686 #39570

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

large number of tracking-related warning/error messages in HLT jobs #39234

large number of tracking-related warning/error messages in HLT jobs #39234

missirol commented Aug 29, 2022

cmsbuild commented Aug 29, 2022

makortel commented Aug 29, 2022

cmsbuild commented Aug 29, 2022

makortel commented Aug 29, 2022

mmusich commented Aug 29, 2022

swagata87 commented Aug 29, 2022

fwyzard commented Aug 29, 2022

slava77 commented Aug 29, 2022

fwyzard commented Aug 29, 2022 via email

swagata87 commented Aug 31, 2022

fwyzard commented Aug 31, 2022

fwyzard commented Aug 31, 2022

mmusich commented Aug 31, 2022

fwyzard commented Aug 31, 2022

fwyzard commented Aug 31, 2022

mmusich commented Aug 31, 2022 •

edited

Loading

missirol commented Sep 5, 2022

swagata87 commented Sep 5, 2022

wonpoint4 commented Oct 3, 2022

missirol commented Oct 3, 2022

missirol commented Oct 4, 2022

missirol commented Oct 14, 2022

mmusich commented Oct 14, 2022

large number of tracking-related warning/error messages in HLT jobs #39234

large number of tracking-related warning/error messages in HLT jobs #39234

Comments

missirol commented Aug 29, 2022

cmsbuild commented Aug 29, 2022

makortel commented Aug 29, 2022

cmsbuild commented Aug 29, 2022

makortel commented Aug 29, 2022

mmusich commented Aug 29, 2022

swagata87 commented Aug 29, 2022

fwyzard commented Aug 29, 2022

slava77 commented Aug 29, 2022

fwyzard commented Aug 29, 2022 via email

swagata87 commented Aug 31, 2022

fwyzard commented Aug 31, 2022

fwyzard commented Aug 31, 2022

mmusich commented Aug 31, 2022

fwyzard commented Aug 31, 2022

fwyzard commented Aug 31, 2022

mmusich commented Aug 31, 2022 • edited Loading

missirol commented Sep 5, 2022

swagata87 commented Sep 5, 2022

wonpoint4 commented Oct 3, 2022

missirol commented Oct 3, 2022

missirol commented Oct 4, 2022

missirol commented Oct 14, 2022

mmusich commented Oct 14, 2022

mmusich commented Aug 31, 2022 •

edited

Loading