Vectorized 2D simulated annealing vertexing

[lgray]

Implement vectorized 2D simulated annealing for vertices with timing.
Use as default vertexing.

Performance measurements in progress.

[cmsbuild]

A new Pull Request was created by @lgray (Lindsey Gray) for master.

It involves the following packages:

RecoVertex/Configuration
RecoVertex/PrimaryVertexProducer

@perrotta, @cmsbuild, @slava77, @davidlange6 can you please review it and eventually sign? Thanks.
@makortel, @felicepantaleo, @GiacomoSguazzoni, @rovere, @VinInn, @ebrondol, @dgulhan this is something you requested to watch as well.
@davidlange6 you are the release manager for this.

cms-bot commands are listed here

[lgray]

@cmsbuild please test

[cmsbuild]

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/21815/console Started: 2017/07/27 13:25

[lgray]

@bendavid FYI

[lgray]

@cmsbuild please abort

[cmsbuild]

Jenkins tests are aborted.

[cmsbuild]

Pull request #19935 was updated. @ghellwig, @ianna, @kpedro88, @fabozzi, @Martin-Grunewald, @rekovic, @vanbesien, @perrotta, @civanch, @silviodonato, @cmsbuild, @fwyzard, @GurpreetSinghChahal, @davidlange6, @smuzaffar, @Dr15Jones, @kkousour, @mdhildreth, @cerminar, @slava77, @ggovi, @mulhearn, @vazzolini, @kmaeshima, @arunhep, @dmitrijus, @alja, @franzoni, @lpernie can you please check and sign again.

[kpedro88]

@slava77 you should be able to see the vertex validation results in WF 20434.0 (2023D19, w/ timing layer)

[cmsbuild]

+1
Tested at: 163dda9
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19935/22871/summary.html

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:
dfc38ed
You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19935/22871/git-log-recent-commits
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19935/22871/git-merge-result

[cmsbuild]

Comparison job queued.

[cmsbuild]

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19935/22871/summary.html

There are some workflows for which there are errors in the baseline:
1330.0 step 4
The results for the comparisons for these workflows could be incomplete
This means most likely that the IB is having errors in the relvals.The error does NOT come from this pull request

Comparison Summary:

• No significant changes to the logs found
• Reco comparison results: 17 differences found in the comparisons
• DQMHistoTests: Total files compared: 26
• DQMHistoTests: Total histograms compared: 2563849
• DQMHistoTests: Total failures: 1014
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 2562649
• DQMHistoTests: Total skipped: 186
• DQMHistoTests: Total Missing objects: 0
• Checked 107 log files, 14 edm output root files, 26 DQM output files

[kpedro88]

+1

[slava77]

Compared to my last checks #19935 (comment)
with the latest version the agreement with the baseline is much better:

There are 2% fewer vertices, but the parameters appear to be much more similar than before

Most relevant diffs since then are:

-        Tpurge = cms.double(3.0),         # cleaning
+        Tpurge = cms.double(4.0),         # cleaning

-  double Tmin = conf.getParameter<double> ("Tmin");
-  double Tpurge = conf.getParameter<double> ("Tpurge");
-  double Tstop = conf.getParameter<double> ("Tstop");
+  double minT = conf.getParameter<double> ("Tmin")*std::sqrt(2.0);
+  double purgeT = conf.getParameter<double> ("Tpurge")*std::sqrt(2.0);
+  double stopT = conf.getParameter<double> ("Tstop")*std::sqrt(2.0);
+  coolingFactor_ = std::sqrt(coolingFactor_);

PV validation also makes sense
Efficiency is ~unchanged

Duplicate rate is down

There are more "pure" vertices (matched and not split or duplicates)

Matched vertices are slightly more separated

Still, there is some loss of response in matched PVs

CPU time is down by a factor of 6 (from 240 to ~40 s/evt).

[slava77]

+1

for #19935 163dda9

• vectorized and retuned ZT DA vertexing, to be used in timing eras for phase-2 (the old non-vector version is removed)
• jenkins tests pass and comparisons show small changes in 4D PV reco plots in wf 20434 (which includes the timing vertices)
• local tests show roughly expected behavior: expected speedup and reasonably similar physics performance as observed in Vectorized 2D simulated annealing vertexing #19935 (comment)

At some point, follow up to this PR is expected following #20417

[lgray]

@vanbesien, @kmaeshima, @vazzolini could one of you please sign for DQM? Thanks!

[lgray]

@slava77 OK looks like there can be further tuning of the clustering parameters to further improve the situation. Similarly, looking how how the 1D annealing was spruced up for Phase 1, I have a few ideas to follow up for this. However, that will come at a later time.

What's here is enough to perform reasonable studies.

[dmitrijus]

+1

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @davidlange6, @slava77, @smuzaffar (and backports should be raised in the release meeting by the corresponding L2)

[davidlange6]

+1