mkfit pixelLess-specific DNN training

[leonardogiannini]

PR description:

This is a follow up PR to #39715, where track selection DNNs were updated. The weights are already included in https://github.com/cms-data/RecoTracker-FinalTrackSelectors

This PR introduces a pixelLess specific DNN with its tuned working points for mkFit workflows. The DNN is not loaded nor called by default as the pixelLess iteration now uses CKF tracking.

No changes are expected when testing the PR.

The PR can be useful for testing the impact of reintroducing the mkFit pixelLess iteration into the default tracking.

PR validation:

the improvements brought to the pixelLess DNN iteration in PU and no PU samples are described in this presentation
https://indico.cern.ch/event/1213630/#3-dnn-tracking-classification

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-39854/32748

• This PR adds an extra 16KB to repository

[cmsbuild]

A new Pull Request was created by @leonardogiannini for master.

It involves the following packages:

• RecoTracker/FinalTrackSelectors (reconstruction)
• RecoTracker/IterativeTracking (reconstruction)

@cmsbuild, @mandrenguyen, @clacaputo can you please review it and eventually sign? Thanks.
@VourMa, @felicepantaleo, @GiacomoSguazzoni, @JanFSchulte, @rovere, @VinInn, @missirol, @ebrondol, @gpetruc, @mmusich, @mtosi, @dgulhan this is something you requested to watch as well.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.

cms-bot commands are listed here

[mmusich]

@cmsbuild, please test

• even though no changes are expected as per mkfit pixelLess-specific DNN training #39854 (comment)

[cmsbuild]

-1

Failed Tests: RelVals-INPUT
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-4a6626/28511/summary.html
COMMIT: e6dbc54
CMSSW: CMSSW_12_6_X_2022-10-25-2300/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/39854/28511/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals-INPUT

The relvals timed out after 4 hours.

Comparison Summary

There are some workflows for which there are errors in the baseline:
20834.911 step 3
4.6 step 3
The results for the comparisons for these workflows could be incomplete
This means most likely that the IB is having errors in the relvals.The error does NOT come from this pull request

Summary:

• No significant changes to the logs found
• Reco comparison results: 266 differences found in the comparisons
• Reco comparison had 6 failed jobs
• DQMHistoTests: Total files compared: 48
• DQMHistoTests: Total histograms compared: 3384029
• DQMHistoTests: Total failures: 93
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3383914
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
• Checked 201 log files, 48 edm output root files, 48 DQM output files
• TriggerResults: no differences found

[slava77]

type tracking

[slava77]

even though no changes are expected as per

11634.7 is supposed to have changes

[mmusich]

11634.7 is supposed to have changes

right, hopefully these are real.

[slava77]

@leonardogiannini
please edit the title of the PR to be more descriptive
e.g. mkfit pixelLess-specific DNN training

[slava77]

@smuzaffar
I see that a few validateJR jobs failed with module-related errors (and apparently some failed comparisons also came out with a difference in the plots)
Was there some change in the related packages that may suggest some incomplete build. Or is this issue otherwise known?

/cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc10/lcg/root/6.24.07-26a448a5bfafa671ac41ad592c8e7413/etc//cling/std.modulemap:368:10: error: module 'std.codecvt' is incompatible with feature 'header_existence'
  module "codecvt" {
         ^
/cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc10/external/gcc/10.3.0-84898dea653199466402e67d73657f10/bin/../lib/gcc/x86_64-redhat-linux-gnu/10.3.0/../../../../include/c++/10.3.0/bits/fs_path.h:40:10: note: submodule of top-level module 'std' implicitly imported here
#include <codecvt>
         ^

[smuzaffar]

@slava77 , these errors were there even before the validateJR.sh update ( see #39735 ). We need to get root fix cms-sw/root#176 in our root 6.24 builds

[slava77]

DQMHistoTests: Total failures: 93

I think that only 11634.7 is expected for this PR; the rest seems unrelated.

Some changes in D88 workflows in JetMET plots appear in other PRs as well e.g. https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-d58b8e/28521/summary.html in #39087 @cms-sw/dqm-l2 @cms-sw/upgrade-l2 @cms-sw/reconstruction-l2 are you familiar with these apparently spurious differences? It's unclear if it's just a feature of CMSSW_12_6_X_2022-10-25-2300
e.g. in 20896 D88 wf

[mmusich]

Some changes in D88 workflows in JetMET plots appear in other PRs as well e.g.

related to #39754 ?

[slava77]

looking at bin-to-bin for wf 11634.7, the comparisons look roughly as expected: changes in tracks start from selected pixelLess tracks
e.g. there are more displaced tracks in the high purity selections of the pixelLess iteration https://tinyurl.com/2bgk7q8e and the 3 extra tracks of all reco are also associated to sim https://tinyurl.com/2ar3vnog

These extra tracks are below 0.9 GeV and don't make it to the efficiency plots vs vtx pos (other eff plots require small displacement)

This is consistent with expectations shown in the POG slides (see the PR description)

[clacaputo]

DQMHistoTests: Total failures: 93

I think that only 11634.7 is expected for this PR; the rest seems unrelated.

Some changes in D88 workflows in JetMET plots appear in other PRs as well e.g. https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-d58b8e/28521/summary.html in #39087 @cms-sw/dqm-l2 @cms-sw/upgrade-l2 @cms-sw/reconstruction-l2 are you familiar with these apparently spurious differences? It's unclear if it's just a feature of CMSSW_12_6_X_2022-10-25-2300 e.g. in 20896 D88 wf

Hi @slava77 , I've seen the same spurious differences in #38784 , that is just a technical PR. As you suggested, it could be a feature of CMSSW_12_6_X_2022-10-25-2300

[mmusich]

@clacaputo are there going to be more comments on the code, or shall we go ahead and restart tests here?

[mmusich]

please test

• to possibly get a cleaner test comparisons

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-4a6626/28656/summary.html
COMMIT: e6dbc54
CMSSW: CMSSW_12_6_X_2022-10-30-2300/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/39854/28656/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 266 differences found in the comparisons
• DQMHistoTests: Total files compared: 48
• DQMHistoTests: Total histograms compared: 3416356
• DQMHistoTests: Total failures: 6
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3416328
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
• Checked 206 log files, 48 edm output root files, 48 DQM output files
• TriggerResults: no differences found

[clacaputo]

+reconstruction

• new pixelLess-specific DNN training for mkFit
  • not called by default
• reco changes only in wf 11634.7, as expected

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

[rappoccio]

+1

• Changes are expected.