New Vertex Clustering in Blocks and Weighted Mean Estimator

[AdrianoDee]

Vast majority of the work from @giorgiopizz.

This PR proposes the addition of a new track clusterizer and a new 3D position estimator for the PrimaryVertexProducer:

• the new clusterizer sorts the tracks in the z coordinate, splits them in blocks of same size (set by default to 512) with a fixed overlap fraction between blocks (set by default to 0.5) and performs independently the Determistin Annealing along all the blocks. The block size and the overlap fraction are made configurable parameters.

• the new estimator iteratively estimates the vertex 3D coordinates and errors using the weighted mean of tracks impact point at the beamspot position and uncertainty. The iterations includes an outlier rejection to improve the performance.

This PR does not introduce the new vertexing as default but add these options to the PrimaryVertexProducer. The new clustering and vertexing are used only in case the (new) processes modifiers vertexInBlocks and weightedVertexing are added to the process. Two workflows are added to the matrix to test this setups: *.278 which runs the full reco with the weighted vertexing and the *.279 which runs trackingOnly wfs. A version of the algorithm for BS constrained vertexing is also added.

Samples for validation

• Phase1: /RelValTTbar_14TeV/CMSSW_12_5_0_pre5-PU_125X_mcRun3_2022_realistic_v3-v2/
• Phase2: /RelValTTbar_14TeV/CMSSW_12_4_0_pre3-PU_123X_mcRun4_realistic_v11_2026D88PU200-v1/

Computational Performance

Here the timing performance for various working points evaluated on fu-c2a02-37-01 machine at P5 (8 threads 8 streams).

PrimaryVertexProducer cpu time for Phase1/Phase2 samples for different block_sizes and overlap_fracs:

PrimaryVertexProducer speed up w.r.t the current producer for Phase1/Phase2 samples for different block_sizes and overlap_fracs:

Physics Performance

• Phase1 validation plots (from MTV), with the overlap_frac=0.5 and block_size=128 here.
• Phase2 validation plots for block_size=128,256,512 (in all plots cpu stands for the current algorithm version):
  • plots for overlap_frac=0.15;
  • plots for overlap_frac=0.25;
  • plots for overlap_frac=0.50.
• Phase2 different contributions: find here a collection of results that shows the different contributions of the new clusterizer and the new estimator altogether and decoupled.

Further references and readings

https://indico.cern.ch/event/1175120/#2-offloading-deterministic-ann
https://indico.cern.ch/event/1206394/#47-dp-note-approval-primary-ve
https://indico.cern.ch/event/1152032/#4-offloading-deterministic-ann

[cmsbuild]

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-39995/32929

• This PR adds an extra 200KB to repository

• There are other open Pull requests which might conflict with changes you have proposed:

  • File Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py modified in PR(s): Add tracking only PU workflows #39722, Add run3 tracking low pu era #33532

Code check has found code style and quality issues which could be resolved by applying following patch(s)

• code-format:
  https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-39995/32929/code-format.patch
  e.g. curl -k https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-39995/32929/code-format.patch | patch -p1
  You can also run scram build code-format to apply code format directly

[mmusich]

type tracking, performance-improvements

[mmusich]

@AdrianoDee it looks like you picked 887ac80 by mistake in this branch (during rebase?)

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-39995/32931

• This PR adds an extra 64KB to repository

• There are other open Pull requests which might conflict with changes you have proposed:

  • File Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py modified in PR(s): Add tracking only PU workflows #39722, Add run3 tracking low pu era #33532

[cmsbuild]

A new Pull Request was created by @AdrianoDee for master.

It involves the following packages:

• Configuration/ProcessModifiers (operations)
• Configuration/PyReleaseValidation (pdmv, upgrade)
• EventFilter/Utilities (daq)
• RecoTracker/IterativeTracking (reconstruction)
• RecoVertex/Configuration (reconstruction)
• RecoVertex/PrimaryVertexProducer (reconstruction)

@perrotta, @rappoccio, @bbilin, @emeschi, @clacaputo, @smorovic, @cmsbuild, @AdrianoDee, @srimanob, @kskovpen, @sunilUIET, @mandrenguyen, @fabiocos, @davidlange6 can you please review it and eventually sign? Thanks.
@beaucero, @fabiocos, @VourMa, @makortel, @felicepantaleo, @GiacomoSguazzoni, @JanFSchulte, @rovere, @VinInn, @Martin-Grunewald, @missirol, @ebrondol, @kpedro88, @trtomei, @gpetruc, @mmusich, @mtosi, @dgulhan, @slomeo this is something you requested to watch as well.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.

cms-bot commands are listed here

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-39995/32932

• This PR adds an extra 64KB to repository

• There are other open Pull requests which might conflict with changes you have proposed:

  • File Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py modified in PR(s): Add tracking only PU workflows #39722, Add run3 tracking low pu era #33532

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ea5456/29268/summary.html
COMMIT: 0178162
CMSSW: CMSSW_12_6_X_2022-11-25-1100/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/39995/29268/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 6 differences found in the comparisons
• DQMHistoTests: Total files compared: 48
• DQMHistoTests: Total histograms compared: 3417239
• DQMHistoTests: Total failures: 6
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3417211
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
• Checked 206 log files, 48 edm output root files, 48 DQM output files
• TriggerResults: no differences found

[AdrianoDee]

+upgrade
(re-sign)

[mmusich]

+1

• re-sign

[mmusich]

@cms-sw/reconstruction-l2 @cms-sw/pdmv-l2 please take a look again. Thx

[mandrenguyen]

+reconstruction
resign

[kskovpen]

+pdmv

[mmusich]

@cms-sw/orp-l2 this is essentially fully-signed

[perrotta]

+1

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will be automatically merged.