Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Backport to CMSSW_13_0_X of new extended ParticleNet networks for jet classification, regression, and taus #41265

Merged
merged 24 commits into from
Apr 12, 2023
Merged

Backport to CMSSW_13_0_X of new extended ParticleNet networks for jet classification, regression, and taus #41265

merged 24 commits into from
Apr 12, 2023

Conversation

scooperstein
Copy link

This pull request would backport the recently merged PR here into CMSSW_13_0_X. This PR would add a variety of new performance improvements in jet flavor classification, jet energy regression, and tau ID and reconstruction that are important additions to include for Run-3 analyses. For many more details, please refer to the original PR. The code changes from the PR itself should be identical to those just merged in the master branch.

Stephane Cooperstein added 24 commits February 10, 2023 10:27
add ParticleNetFromMiniAOD
17471e8
a few fixes after runtime testing
30c3043
remove commented line
419dc21
reformat new code
5c0afbd
add test cfgs for ParticleNetFromMiniAOD
0659445
first implementation of new extended ParticleNet networks in miniAOD …
17457a5 
…and nanoAOD
clean up saved output nodes
06e0181
fix reference to PNet miniAOD tags
da0325f
remove DeepAK8 from miniAOD and nanoAOD, include full ParticleNet mas…
7e5767b 
…s correlated outputs
enable evaluation of new ParticleNet trainings in nanoAOD step
8903fcd
fix printouts
ac075a6
add PNet regression neutrino correction
3bca26d
fix jet and JEC reference in nanoAOD PNet reevaluation for PUPPI
57484c4
remove puppi weight from PNet feature evaluator and fix nano AK4 PUPP…
5e7f771 
…I jet config
fix CvsL definition
69a2c39
add PNet to NanoAOD DQM and change name of mass-correlated tagger nodes
acbdc84
save decommissioned taggers for older Nano eras and infer new PNet no…
1ecd5da 
…des for Run-2 UL
also remove new PNet AK4 nodes for PUPPI jets eras 12_2/4
71d3b02
remove extra space
4ceeb26
update DQM plots
525aadc
don't run PNet AK4 on PUPPI for Run-2
61e61f4
fix DQM variables with ParticleNet
0075526
remove duplicate variable
d524a88
add missing commas
3c15acd
@cmsbuild cmsbuild added this to the CMSSW_13_0_X milestone Apr 4, 2023
@perrotta
Copy link
Contributor

tests seem stuck: try to restart them

@perrotta
Copy link
Contributor

test parameters:

@perrotta
Copy link
Contributor

cms-sw/cmsdist#8436 backports cms-sw/cmsdist#8422 to 13_0_X

@perrotta
Copy link
Contributor

please test

@cmsbuild
Copy link
Contributor

-1

Failed Tests: RelVals-INPUT
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-97d501/31909/summary.html
COMMIT: 3c15acd
CMSSW: CMSSW_13_0_X_2023-04-11-1100/el8_amd64_gcc11
Additional Tests: NANO
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/41265/31909/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals-INPUT

  • 140.6140.6_RunHI2022/step2_RunHI2022.log
  • 1002.41002.4_RunDoubleMuon2022B/step2_RunDoubleMuon2022B.log
  • 11834.9911834.99_TTbar_14TeV+2021PU_PMXS1S2/step2_TTbar_14TeV+2021PU_PMXS1S2.log
Expand to see more relval errors ...

Comparison Summary

Summary:

  • You potentially added 27 lines to the logs
  • ROOTFileChecks: Some differences in event products or their sizes found
  • Reco comparison results: 6154 differences found in the comparisons
  • DQMHistoTests: Total files compared: 49
  • DQMHistoTests: Total histograms compared: 3553808
  • DQMHistoTests: Total failures: 106
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 3553680
  • DQMHistoTests: Total skipped: 22
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 45.801 KiB( 48 files compared)
  • DQMHistoSizes: changed ( 11634.0,... ): 5.549 KiB Physics/NanoAODDQM
  • DQMHistoSizes: changed ( 13234.0,... ): 3.479 KiB Physics/NanoAODDQM
  • Checked 213 log files, 164 edm output root files, 49 DQM output files
  • TriggerResults: no differences found

NANO Comparison Summary

Summary:

  • You potentially added 51 lines to the logs
  • ROOTFileChecks: Some differences in event products or their sizes found
  • Reco comparison results: 152 differences found in the comparisons
  • DQMHistoTests: Total files compared: 12
  • DQMHistoTests: Total histograms compared: 11636
  • DQMHistoTests: Total failures: 173
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 11463
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 51.254 KiB( 11 files compared)
  • DQMHistoSizes: changed ( 2500.311,... ): 8.261 KiB Physics/NanoAODDQM
  • DQMHistoSizes: changed ( 2500.331,... ): 6.190 KiB Physics/NanoAODDQM
  • DQMHistoSizes: changed ( 2500.401,... ): 0.588 KiB Physics/NanoAODDQM
  • DQMHistoSizes: changed ( 2500.601 ): 5.549 KiB Physics/NanoAODDQM
  • Checked 25 log files, 11 edm output root files, 12 DQM output files

Nano size comparison Summary:

Sample kb/ev ref kb/ev diff kb/ev ev/s/thd ref ev/s/thd diff rate mem/thd ref mem/thd
2500.31 2.343 2.232 0.110 ( +4.9% ) 5.58 9.41 -40.7% 1.420 1.390
2500.311 2.440 2.322 0.118 ( +5.1% ) 5.11 9.04 -43.5% 1.781 1.741
2500.312 2.391 2.275 0.116 ( +5.1% ) 5.25 9.23 -43.1% 1.771 1.732
2500.33 1.172 1.098 0.074 ( +6.7% ) 10.61 21.16 -49.9% 1.484 1.617
2500.331 1.508 1.392 0.115 ( +8.3% ) 5.83 15.30 -61.9% 1.651 1.612
2500.332 1.409 1.324 0.086 ( +6.5% ) 8.29 17.20 -51.8% 1.593 1.551
2500.401 2.137 2.137 0.000 ( +0.0% ) 10.45 10.39 +0.5% 1.106 1.099
2500.501 1.710 1.710 -0.000 ( -0.0% ) 16.91 16.75 +1.0% 1.057 1.063
2500.511 1.122 1.122 0.000 ( +0.0% ) 30.98 30.63 +1.1% 1.318 1.307
2500.5111 1.477 1.477 0.000 ( +0.0% ) 19.85 16.82 +18.0% 1.363 1.362
2500.601 2.038 2.049 -0.011 ( -0.5% ) 12.58 12.52 +0.4% 1.085 1.087

@simonepigazzini
Copy link
Contributor

I'm not sure of what is causing the failures. The changes in the PR should not affect those workflows.

@perrotta
Copy link
Contributor

perrotta commented Apr 12, 2023 •
edited
Loading

@simonepigazzini timeouts in RelValInputs workflows are quite common
Let retest, to see if we can get rid of them
(Otherwise, just ignore them)

@perrotta
Copy link
Contributor

please test

@cmsbuild
Copy link
Contributor

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-97d501/31930/summary.html
COMMIT: 3c15acd
CMSSW: CMSSW_13_0_X_2023-04-11-2300/el8_amd64_gcc11
Additional Tests: NANO
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/41265/31930/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-97d501/31930/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-97d501/31930/git-merge-result

Comparison Summary

Summary:

  • You potentially removed 12 lines from the logs
  • ROOTFileChecks: Some differences in event products or their sizes found
  • Reco comparison results: 6155 differences found in the comparisons
  • DQMHistoTests: Total files compared: 49
  • DQMHistoTests: Total histograms compared: 3553808
  • DQMHistoTests: Total failures: 103
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 3553683
  • DQMHistoTests: Total skipped: 22
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 45.801 KiB( 48 files compared)
  • DQMHistoSizes: changed ( 11634.0,... ): 5.549 KiB Physics/NanoAODDQM
  • DQMHistoSizes: changed ( 13234.0,... ): 3.479 KiB Physics/NanoAODDQM
  • Checked 213 log files, 164 edm output root files, 49 DQM output files
  • TriggerResults: no differences found

NANO Comparison Summary

Summary:

  • You potentially added 59 lines to the logs
  • ROOTFileChecks: Some differences in event products or their sizes found
  • Reco comparison results: 152 differences found in the comparisons
  • DQMHistoTests: Total files compared: 12
  • DQMHistoTests: Total histograms compared: 11636
  • DQMHistoTests: Total failures: 173
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 11463
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 51.254 KiB( 11 files compared)
  • DQMHistoSizes: changed ( 2500.311,... ): 8.261 KiB Physics/NanoAODDQM
  • DQMHistoSizes: changed ( 2500.331,... ): 6.190 KiB Physics/NanoAODDQM
  • DQMHistoSizes: changed ( 2500.401,... ): 0.588 KiB Physics/NanoAODDQM
  • DQMHistoSizes: changed ( 2500.601 ): 5.549 KiB Physics/NanoAODDQM
  • Checked 25 log files, 11 edm output root files, 12 DQM output files

Nano size comparison Summary:

Sample kb/ev ref kb/ev diff kb/ev ev/s/thd ref ev/s/thd diff rate mem/thd ref mem/thd
2500.31 2.343 2.232 0.110 ( +4.9% ) 5.59 9.42 -40.6% 1.415 1.458
2500.311 2.440 2.322 0.118 ( +5.1% ) 5.01 9.01 -44.4% 1.771 1.747
2500.312 2.391 2.275 0.116 ( +5.1% ) 5.23 9.12 -42.7% 1.765 1.738
2500.33 1.172 1.098 0.074 ( +6.7% ) 10.54 21.09 -50.0% 1.485 1.622
2500.331 1.508 1.392 0.115 ( +8.3% ) 5.76 15.35 -62.5% 1.650 1.610
2500.332 1.409 1.324 0.086 ( +6.5% ) 8.20 17.28 -52.5% 1.585 1.575
2500.401 2.137 2.137 0.000 ( +0.0% ) 10.35 10.42 -0.7% 1.092 1.181
2500.501 1.710 1.710 -0.000 ( -0.0% ) 16.59 16.69 -0.6% 1.046 1.121
2500.511 1.122 1.122 0.000 ( +0.0% ) 30.67 30.68 -0.1% 1.289 1.364
2500.5111 1.477 1.477 0.000 ( +0.0% ) 19.05 19.47 -2.1% 1.335 1.408
2500.601 2.038 2.049 -0.011 ( -0.5% ) 12.49 12.27 +1.8% 1.079 1.104

@simonepigazzini
Copy link
Contributor

+1

The changes are in line with the original PR. Signing this off from xpog

@mandrenguyen
Copy link
Contributor

+1

@cmsbuild
Copy link
Contributor

This pull request is fully signed and it will be integrated in one of the next CMSSW_13_0_X IBs (tests are also fine) and once validation in the development release cycle CMSSW_13_1_X is complete. This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

@perrotta
Copy link
Contributor

+1

  • Only mini and nano involved with the addition of ParticleNet, ok to backport

@cmsbuild cmsbuild merged commit ef63447 into cms-sw:CMSSW_13_0_X Apr 12, 2023
@slava77 slava77 mentioned this pull request May 2, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
backport-ok fully-signed orp-approved reconstruction-approved requires-external tests-approved xpog-approved
Projects
None yet
Milestone
CMSSW_13_0_X
Development

Successfully merging this pull request may close these issues.
6 participants
@scooperstein @cmsbuild @simonepigazzini @swertz @mandrenguyen @perrotta