PFID with DNNs - updated model trained on Run3Summer21 #36243

valsdav · 2021-11-24T21:52:09Z

PR description:

This PR updates the PFID DNN models trained on Run3Summer21 samples. A few additional variables have been included for electrons.
The paths of the models have been updated as well as the thresholds defining the default working point.
An additional threshold on the non-isolated background class has been included for electrons in PFEgammaFilters.

This PR comes along two PRs for ElectronIdentification (cms-data/RecoEgamma-ElectronIdentification#24) and PhotonIdentification (cms-data/RecoEgamma-PhotonIdentification#10) data repositories.

PR validation:

For documentation about the changes in the model training and input variables please refer to the slides at: https://indico.cern.ch/event/1096395/#12-egm-updates-for-run-3-prepa.

The correctness of the DNN configuration has been verified. A validation of the physics performance of the PFID WPs is ongoing in the PF and Egamma group.

cmsbuild · 2021-11-24T21:59:46Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-36243/26878

This PR adds an extra 40KB to repository

cmsbuild · 2021-11-24T22:00:06Z

A new Pull Request was created by @valsdav (Davide Valsecchi) for master.

It involves the following packages:

RecoEgamma/EgammaElectronProducers (reconstruction)
RecoEgamma/EgammaPhotonProducers (reconstruction)
RecoEgamma/ElectronIdentification (reconstruction)
RecoEgamma/PhotonIdentification (reconstruction)
RecoParticleFlow/PFProducer (reconstruction)

@jpata, @cmsbuild, @slava77 can you please review it and eventually sign? Thanks.
@mmarionncern, @Sam-Harper, @afiqaize, @jainshilpi, @cbernet, @rovere, @lgray, @sobhatta, @lecriste, @hatakeyamak, @wrtabb, @varuns23, @seemasharmafnal, @ram1123 this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

RecoEgamma/ElectronIdentification/src/ElectronDNNEstimator.cc

cmsbuild · 2021-11-25T09:43:30Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-36243/26893

This PR adds an extra 40KB to repository

cmsbuild · 2021-11-25T09:43:57Z

Pull request #36243 was updated. @jpata, @cmsbuild, @slava77 can you please check and sign again.

perrotta · 2021-11-25T09:40:51Z

RecoParticleFlow/PFProducer/interface/PFEGammaFilters.h

@@ -57,6 +57,10 @@ class PFEGammaFilters {
  float ele_dnnLowPtThr_;
  float ele_dnnHighPtBarrelThr_;
  float ele_dnnHighPtEndcapThr_;


All these *Thr_ , as well as the ele_max* and most of the badHcal_* ones, can be made const

We kept them non-const only because they are read from a group of options in the constructor https://github.com/cms-sw/cmssw/blob/master/RecoParticleFlow/PFProducer/src/PFEGammaFilters.cc#L72 and I felt it would be ugly to put that in the initializer list. How do you suggest to fix this problem? Thanks

Uhm, yes, it would require some gymnastic indeed...
Ok, don't mind

cmsbuild · 2021-11-25T10:42:12Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-36243/26894

This PR adds an extra 40KB to repository

cmsbuild · 2021-11-25T10:42:38Z

Pull request #36243 was updated. @jpata, @cmsbuild, @slava77 can you please check and sign again.

slava77 · 2021-11-25T14:31:59Z

test parameters:

pull_requests = Updated PFID DNN model files trained for Run3Summer21 cms-data/RecoEgamma-ElectronIdentification#24,Updated PFID DNN model files trained for Run3Summer21 cms-data/RecoEgamma-PhotonIdentification#10

jpata

minor comment inline. let's run a test with profiling too.

jpata · 2021-11-26T12:54:23Z

RecoParticleFlow/PFProducer/python/particleFlow_cff.py

@@ -15,14 +15,22 @@
 particleFlowTmp.PFEGammaFiltersParameters.allowEEEinPF = cms.bool(False)

 # Thresholds for e/gamma PFID DNN 
+# Thresholds for electron: Sig_isolated+Sig_nonIsolated


might be good to have a reference to an indico presentation here with these numbers

Hi @jpata, It should be this one https://indico.cern.ch/event/1096395/#12-egm-updates-for-run-3-prepa., shown at the last PPD meeting by @akapoorcern.

right, please put it in the code comment :)

jpata · 2021-11-26T12:56:35Z

@cmsbuild please test

cmsbuild · 2021-11-27T00:46:16Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-d5bc9c/20787/summary.html
COMMIT: 2325db7
CMSSW: CMSSW_12_2_X_2021-11-26-1100/slc7_amd64_gcc900
Additional Tests: PROFILING
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/36243/20787/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 8313 differences found in the comparisons
DQMHistoTests: Total files compared: 42
DQMHistoTests: Total histograms compared: 3247873
DQMHistoTests: Total failures: 8484
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3239367
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 41 files compared)
Checked 177 log files, 37 edm output root files, 42 DQM output files
TriggerResults: no differences found

jpata · 2021-11-27T08:43:46Z

@valsdav please summarize the CPU and MEM changes of this PR (initialization and processing) based on the profiles

jpata · 2021-11-29T09:01:44Z

memory in initialization
- baseline
- this PR
memory in processing
- baseline
- this PR
CPU in processing
- baseline
- this PR

jpata · 2021-11-29T09:08:26Z

+reconstruction

new Run3 training for PFID, required for the next 12_2_0 release
reco changes expected and observed in quantities downstream from PF (PF candidates, PF jets and MET, GSF electrons, GED photons, taus)
according to PFID with DNNs - updated model trained on Run3Summer21 #36243 (comment), we don't observe any significant change in Run3, step3 memory or CPU consumption
please merge concurrently with the data PRs Updated PFID DNN model files trained for Run3Summer21 cms-data/RecoEgamma-ElectronIdentification#24, Updated PFID DNN model files trained for Run3Summer21 cms-data/RecoEgamma-PhotonIdentification#10

cmsbuild · 2021-11-29T09:08:51Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

valsdav · 2021-11-29T09:15:39Z

memory in initialization

* baseline ![Screenshot from 2021-11-29 10-53-40](https://user-images.githubusercontent.com/69717/143836977-379a4f40-839a-40ef-bc41-700afb796826.png)
* this PR ![Screenshot from 2021-11-29 10-53-58](https://user-images.githubusercontent.com/69717/143836915-36b8c66d-213e-4e9e-91bc-0bd193e025eb.png)

memory in processing
- baseline
- this PR
CPU in processing
- baseline
- this PR

Thanks @jpata for the summary.

jpata · 2021-11-29T10:10:24Z

this is showing just the ::stream modules. The relevant mkFit modules are ::global

Is mkFit here a typo, or was this comment meant for the mkFit PR? GsfElectronProducer is showing up among the stream modules, and in any case, the total CPU/mem looks also fine on this PR.

perrotta · 2021-11-29T16:03:13Z

+1

To be merged together with Updated PFID DNN model files trained for Run3Summer21 cms-data/RecoEgamma-ElectronIdentification#24 and Updated PFID DNN model files trained for Run3Summer21 cms-data/RecoEgamma-ElectronIdentification#24

valsdav added 3 commits November 15, 2021 17:21

Added new bkg_nonIsolated threshold for PFEgm ID

31eeb60

Added more variables for Electron PFID DNN

3d2cf7f

Model files and thresholds for PFID with DNN trained in Run3Summer21

10c3665

cmsbuild added this to the CMSSW_12_2_X milestone Nov 24, 2021

cmsbuild added code-checks-pending orp-pending pending-signatures reconstruction-pending tests-pending labels Nov 24, 2021

cmsbuild added code-checks-approved and removed code-checks-pending labels Nov 24, 2021

swagata87 reviewed Nov 25, 2021

View reviewed changes

RecoEgamma/ElectronIdentification/src/ElectronDNNEstimator.cc Outdated Show resolved Hide resolved

cmsbuild added code-checks-pending and removed code-checks-approved labels Nov 25, 2021

cmsbuild added code-checks-approved and removed code-checks-pending labels Nov 25, 2021

perrotta reviewed Nov 25, 2021

View reviewed changes

Fixed ElectronDNNEstimator gsfTrack variables

2325db7

valsdav force-pushed the pfid_egamma_v2 branch from cc796ad to 2325db7 Compare November 25, 2021 10:32

cmsbuild added code-checks-pending and removed code-checks-approved labels Nov 25, 2021

cmsbuild added code-checks-approved and removed code-checks-pending labels Nov 25, 2021

cmsbuild added tests-approved and removed tests-started labels Nov 25, 2021

jpata reviewed Nov 26, 2021

View reviewed changes

cmsbuild added tests-started and removed tests-approved labels Nov 26, 2021

cmsbuild added tests-approved and removed tests-started labels Nov 27, 2021

This was referenced Nov 29, 2021

Updated PFID DNN model files trained for Run3Summer21 cms-data/RecoEgamma-ElectronIdentification#24

Merged

Updated PFID DNN model files trained for Run3Summer21 cms-data/RecoEgamma-PhotonIdentification#10

Merged

cmsbuild added fully-signed reconstruction-approved and removed reconstruction-pending pending-signatures labels Nov 29, 2021

cmsbuild added orp-approved and removed orp-pending labels Nov 29, 2021

cmsbuild merged commit e7954c1 into cms-sw:master Nov 29, 2021

This was referenced Nov 29, 2021

Update tag for RecoEgamma-PhotonIdentification to V01-03-00 cms-sw/cmsdist#7477

Merged

Update tag for RecoEgamma-ElectronIdentification to V01-09-00 cms-sw/cmsdist#7478

Merged

valsdav mentioned this pull request Nov 30, 2021

Bugfix for Egamma PFID with DNNs #36292

Merged

perrotta mentioned this pull request Dec 3, 2021

trackingOnly relval matrix workflows looks messed up #36347

Closed

a-kapoor mentioned this pull request May 10, 2022

Particle flow identification of electrons in extended eta #37880

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PFID with DNNs - updated model trained on Run3Summer21 #36243

PFID with DNNs - updated model trained on Run3Summer21 #36243

valsdav commented Nov 24, 2021

cmsbuild commented Nov 24, 2021

cmsbuild commented Nov 24, 2021

cmsbuild commented Nov 25, 2021

cmsbuild commented Nov 25, 2021

perrotta Nov 25, 2021

valsdav Nov 25, 2021

perrotta Nov 25, 2021

cmsbuild commented Nov 25, 2021

cmsbuild commented Nov 25, 2021

slava77 commented Nov 25, 2021

jpata left a comment

jpata Nov 26, 2021

valsdav Nov 26, 2021

jpata Nov 26, 2021

valsdav Nov 26, 2021

jpata commented Nov 26, 2021

cmsbuild commented Nov 27, 2021

jpata commented Nov 27, 2021 •

edited

Loading

jpata commented Nov 29, 2021

jpata commented Nov 29, 2021

cmsbuild commented Nov 29, 2021

valsdav commented Nov 29, 2021

jpata commented Nov 29, 2021

perrotta commented Nov 29, 2021

PFID with DNNs - updated model trained on Run3Summer21 #36243

PFID with DNNs - updated model trained on Run3Summer21 #36243

Conversation

valsdav commented Nov 24, 2021

PR description:

PR validation:

cmsbuild commented Nov 24, 2021

cmsbuild commented Nov 24, 2021

cmsbuild commented Nov 25, 2021

cmsbuild commented Nov 25, 2021

perrotta Nov 25, 2021

Choose a reason for hiding this comment

valsdav Nov 25, 2021

Choose a reason for hiding this comment

perrotta Nov 25, 2021

Choose a reason for hiding this comment

cmsbuild commented Nov 25, 2021

cmsbuild commented Nov 25, 2021

slava77 commented Nov 25, 2021

jpata left a comment

Choose a reason for hiding this comment

jpata Nov 26, 2021

Choose a reason for hiding this comment

valsdav Nov 26, 2021

Choose a reason for hiding this comment

jpata Nov 26, 2021

Choose a reason for hiding this comment

valsdav Nov 26, 2021

Choose a reason for hiding this comment

jpata commented Nov 26, 2021

cmsbuild commented Nov 27, 2021

Comparison Summary

jpata commented Nov 27, 2021 • edited Loading

jpata commented Nov 29, 2021

jpata commented Nov 29, 2021

cmsbuild commented Nov 29, 2021

valsdav commented Nov 29, 2021

jpata commented Nov 29, 2021

perrotta commented Nov 29, 2021

jpata commented Nov 27, 2021 •

edited

Loading