Adjust collections of Tracker clusters to be saved for Phase-2 ALCARECO #40323

mmusich · 2022-12-15T11:07:28Z

PR description:

This is a follow-up of PR #39858.
After the backport of that PR (#39898) was integrated, it was possible to test the production of RelVal samples (see e.g. on DAS: /RelValMinBias_14TeV/CMSSW_12_5_2-TkAlMinBias-125X_mcRun4_realistic_v3_2026D88PU-v1/ALCARECO).
When trying to run over it, it became clear that is not possible to refit tracks, due to the missing event products (the Phase-2 Outer Tracker clusters) that are necessary for CPE re-computation, leading to this error:

Begin processing the 1st record. Run 1, Event 301, LumiSection 4 on stream 6 at 13-Dec-2022 12:50:01.219 CET
----- Begin Fatal Exception 13-Dec-2022 12:50:57 CET-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 1 lumi: 4 event: 306 stream: 2
   [1] Running path 'p2'
   [2] Calling method for module TrackRefitter/'FinalTrackRefitter'
Exception Message:
RefCore: A request to resolve a reference to a product of type 'edmNew::DetSetVector<Phase2TrackerCluster1D>' with ProductID '4:449'
can not be satisfied because the product cannot be found.
Probably the branch containing the product is not stored in the input file.
   Additional Info:
      [a] If you wish to continue processing events after a ProductNotFound exception,
add "SkipEvent = cms.untracked.vstring('ProductNotFound')" to the "options" PSet in the configuration.

----- End Fatal Exception -------------------------------------------------

This PR adds the minimal amount of changes in the CommonTools/RecoAlgos package in order to be able to retain the Phase-2 OT clusters after running the AlignmentTrackSelector module that relies on it:

cmssw/Alignment/CommonAlignmentProducer/plugins/AlignmentTrackSelectorModule.cc

Lines 13 to 17 in 52c0dac

    
           // the following include is necessary to clone all track branches 
        
           // including recoTrackExtras and TrackingRecHitsOwned (in future also "owned clusters"?). 
        
           // if you remove it the code will compile, but the cloned 
        
           // tracks have only the recoTracks branch! 
        
           #include "CommonTools/RecoAlgos/interface/TrackSelector.h"

This is done in commit f5efa50.
Secondly the outputCommands of all the Tracker Alignment ALCARECO producers is modified in order to retain Phase2TrackerCluster1DedmNewDetSetVector in the output - in case the production is run for Phase-2 - while removing SiStripClusteredmNewDetSetVector (since there is no SiStrip detector in phase-2). I profit of that to re-adjust the commands for event contents to cope with no SCAL in Run3 that were leftover from #38415 and #38590.
This is done in commit 7ef7e84.

PR validation:

Run two typical workflows for Run-3 and Phase-2 with

runTheMatrix.py -l 11634.0,20834.0 -t 4 -j 8 --ibeos

And tested the event content of TkAlMinBias indeed changes as expected.

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

Not a backport, but to be backported down to 12.5.X in order to be used for the production happening in that cycle for L1T studies.

…to store the Phase-2 clusters

mmusich · 2022-12-15T11:07:47Z

FYI: @consuegs @antoniovagnerini

mmusich · 2022-12-15T11:08:44Z

type trk

cmsbuild · 2022-12-15T11:16:21Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-40323/33405

This PR adds an extra 32KB to repository

cmsbuild · 2022-12-15T11:16:46Z

A new Pull Request was created by @mmusich (Marco Musich) for master.

It involves the following packages:

Alignment/CommonAlignmentProducer (alca)
CommonTools/RecoAlgos (reconstruction)
DataFormats/TrackerRecHit2D (reconstruction)

@malbouis, @yuanchao, @ChrisMisan, @clacaputo, @cmsbuild, @saumyaphor4252, @tvami, @mandrenguyen, @francescobrivio can you please review it and eventually sign? Thanks.
@pakhotin, @rappoccio, @jdolen, @tlampen, @VourMa, @adewit, @ahinzmann, @abbiendi, @JanFSchulte, @rovere, @VinInn, @missirol, @gkasieczka, @tocheng, @hatakeyamak, @gpetruc, @mmusich, @mtosi, @jhgoh this is something you requested to watch as well.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.

cms-bot commands are listed here

mmusich · 2022-12-15T11:17:09Z

@cmsbuild, please test

cmsbuild · 2022-12-15T12:52:39Z

-1

Failed Tests: RelVals RelVals-INPUT
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1f07ce/29630/summary.html
COMMIT: 7ef7e84
CMSSW: CMSSW_13_0_X_2022-12-14-2300/el8_amd64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/40323/29630/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals

138.4138.4_PromptCollisions2021/step2_PromptCollisions2021.log
138.5138.5_ExpressCollisions2021/step2_ExpressCollisions2021.log
140.53140.53_RunHI2011/step2_RunHI2011.log

Expand to see more relval errors ...

RelVals-INPUT

4.64.6_MinimumBias2010A/step2_MinimumBias2010A.log
138.4138.4_PromptCollisions2021/step2_PromptCollisions2021.log
138.5138.5_ExpressCollisions2021/step2_ExpressCollisions2021.log

Expand to see more relval errors ...

cmsbuild · 2022-12-15T13:05:01Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-40323/33409

This PR adds an extra 32KB to repository

cmsbuild · 2022-12-15T13:05:23Z

Pull request #40323 was updated. @malbouis, @yuanchao, @ChrisMisan, @clacaputo, @cmsbuild, @saumyaphor4252, @tvami, @mandrenguyen, @francescobrivio can you please check and sign again.

mmusich · 2022-12-15T21:08:51Z

@cmsbuild, please test

let's try again

cmsbuild · 2022-12-16T01:18:19Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1f07ce/29644/summary.html
COMMIT: a8ed040
CMSSW: CMSSW_13_0_X_2022-12-15-1100/el8_amd64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/40323/29644/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 14 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3557521
DQMHistoTests: Total failures: 12
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3557487
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
Checked 211 log files, 162 edm output root files, 49 DQM output files
TriggerResults: no differences found

mandrenguyen · 2022-12-16T06:19:16Z

+reconstruction
No change to reco

mmusich · 2022-12-16T15:42:41Z

@cms-sw/alca-l2 do you have any objections?
I'd like to backport this for the phase-2 production happening in 12_5_X.

tvami · 2022-12-16T15:50:56Z

+alca

diffs in MsgLogger only

cmsbuild · 2022-12-16T15:51:17Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

perrotta · 2022-12-17T07:54:48Z

+1

perrotta · 2022-12-19T05:54:00Z

@mmusich the situation with the IGs was a bit messy over the weekend. Now that it settled down, we remain with a crash in the Phase2 wf 20834.9 for the AlignmentTrackSelectorModule:ALCARECOMuAlOverlapsGeneralTracks module. It started with CMSSW_13_0_X_2022-12-17-1100, and I suspect it could come from the changes that were implemented with this PR: could you please have a look?

mmusich · 2022-12-19T13:05:07Z

@perrotta

I suspect it could come from the changes that were implemented with this PR

yes, it definitely comes from this PR.
A very simple fix (which is actually a workaround) is to comment this line:

cmssw/CommonTools/RecoAlgos/src/ClusterStorer.cc

Line 53 in 9318eb0

    
           phase2OTClusterRecords_.push_back(Phase2OTClusterHitRecord(static_cast<VectorHit &>(newHit), hits, index));

This will solve the failure in the workflow, but then the AlCa step for the vectorHits workflows still won't do the right thing (at least it is is not worse than it was before this PR). I am working on a more refined fix, but I cannot guarantee it will come before the EOY break. Let me know if this is acceptable from your point of view. From TRK point of view since VH are not the default this would still meet the goal of this PR.

mmusich · 2022-12-19T16:08:17Z

I am working on a more refined fix, but I cannot guarantee it will come before the EOY break. Let me know if this is acceptable from your point of view. From TRK point of view since VH are not the default this would still meet the goal of this PR.

nevermind, see #40363

perrotta · 2022-12-19T17:33:29Z

nevermind, see #40363

Thank you @mmusich !

Introduce necessary modifications to AlignmentTrackSelector in order …

f5efa50

…to store the Phase-2 clusters

cmsbuild added this to the CMSSW_13_0_X milestone Dec 15, 2022

cmsbuild added alca-pending code-checks-pending orp-pending pending-signatures reconstruction-pending tests-pending labels Dec 15, 2022

cmsbuild added the trk label Dec 15, 2022

cmsbuild added code-checks-approved and removed code-checks-pending labels Dec 15, 2022

cmsbuild added tests-started and removed tests-pending labels Dec 15, 2022

cmsbuild added tests-rejected and removed tests-started labels Dec 15, 2022

mmusich force-pushed the AlignmentTrackSelectorPhase2 branch from 7ef7e84 to 5e40b93 Compare December 15, 2022 12:57

cmsbuild added code-checks-pending tests-pending and removed tests-rejected code-checks-approved labels Dec 15, 2022

cmsbuild added code-checks-approved and removed code-checks-pending labels Dec 15, 2022

cmsbuild added tests-started and removed tests-rejected labels Dec 15, 2022

cmsbuild added tests-approved and removed tests-started labels Dec 16, 2022

cmsbuild added reconstruction-approved and removed reconstruction-pending labels Dec 16, 2022

cmsbuild added alca-approved fully-signed and removed alca-pending pending-signatures labels Dec 16, 2022

cmsbuild added orp-approved and removed orp-pending labels Dec 17, 2022

cmsbuild merged commit f9aaf6a into cms-sw:master Dec 17, 2022

mmusich mentioned this pull request Dec 19, 2022

Fix ClusterStorer for VectorHits #40363

Merged

mmusich deleted the AlignmentTrackSelectorPhase2 branch December 19, 2022 16:07

This was referenced Dec 22, 2022

[12.6.X] Adjust collections of Tracker clusters to be saved for Phase-2 ALCARECO #40394

Merged

[12.5.X] Adjust collections of Tracker clusters to be saved for Phase-2 ALCARECO #40395

Merged

mmusich mentioned this pull request Jan 16, 2023

apply necessary changes to test the phase-2 AlCaReco workflow #40542

Merged

mmusich mentioned this pull request Jan 24, 2023

get rid of Phase2TrackerRecHit1D::CluRef #40601

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adjust collections of Tracker clusters to be saved for Phase-2 ALCARECO #40323

Adjust collections of Tracker clusters to be saved for Phase-2 ALCARECO #40323

mmusich commented Dec 15, 2022 •

edited

Loading

mmusich commented Dec 15, 2022

mmusich commented Dec 15, 2022

cmsbuild commented Dec 15, 2022

cmsbuild commented Dec 15, 2022

mmusich commented Dec 15, 2022

cmsbuild commented Dec 15, 2022

cmsbuild commented Dec 15, 2022

cmsbuild commented Dec 15, 2022

mmusich commented Dec 15, 2022

cmsbuild commented Dec 16, 2022

mandrenguyen commented Dec 16, 2022

mmusich commented Dec 16, 2022

tvami commented Dec 16, 2022 •

edited

Loading

cmsbuild commented Dec 16, 2022

perrotta commented Dec 17, 2022

perrotta commented Dec 19, 2022 •

edited

Loading

mmusich commented Dec 19, 2022

mmusich commented Dec 19, 2022

perrotta commented Dec 19, 2022

	// the following include is necessary to clone all track branches
	// including recoTrackExtras and TrackingRecHitsOwned (in future also "owned clusters"?).
	// if you remove it the code will compile, but the cloned
	// tracks have only the recoTracks branch!
	#include "CommonTools/RecoAlgos/interface/TrackSelector.h"

Adjust collections of Tracker clusters to be saved for Phase-2 ALCARECO #40323

Adjust collections of Tracker clusters to be saved for Phase-2 ALCARECO #40323

Conversation

mmusich commented Dec 15, 2022 • edited Loading

PR description:

PR validation:

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

mmusich commented Dec 15, 2022

mmusich commented Dec 15, 2022

cmsbuild commented Dec 15, 2022

cmsbuild commented Dec 15, 2022

mmusich commented Dec 15, 2022

cmsbuild commented Dec 15, 2022

RelVals

RelVals-INPUT

cmsbuild commented Dec 15, 2022

cmsbuild commented Dec 15, 2022

mmusich commented Dec 15, 2022

cmsbuild commented Dec 16, 2022

Comparison Summary

mandrenguyen commented Dec 16, 2022

mmusich commented Dec 16, 2022

tvami commented Dec 16, 2022 • edited Loading

cmsbuild commented Dec 16, 2022

perrotta commented Dec 17, 2022

perrotta commented Dec 19, 2022 • edited Loading

mmusich commented Dec 19, 2022

mmusich commented Dec 19, 2022

perrotta commented Dec 19, 2022

mmusich commented Dec 15, 2022 •

edited

Loading

tvami commented Dec 16, 2022 •

edited

Loading

perrotta commented Dec 19, 2022 •

edited

Loading