Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[NanoAOD] Add ParticleNet-based Muon ID (Rework of #46153) #46273

Merged
merged 6 commits into from
Oct 31, 2024

Conversation

hqucms
Copy link
Contributor

@hqucms hqucms commented Oct 4, 2024

PR description:

Rework of #46153.
The updated model file is in cms-data/PhysicsTools-NanoAOD#18.

PR description:

This PR adds ParticleNet-based Muon ID for NanoAODv15.
The feature was discussed during Muon POG meeting (23/09/2024) and it will be discussed during XPOG meeting (2/10/2024)

PR validation:

runTheMatrix.py -w nano --ibeos -l 2500.211

@cmsbuild
Copy link
Contributor

cmsbuild commented Oct 4, 2024

cms-bot internal usage

@hqucms
Copy link
Contributor Author

hqucms commented Oct 4, 2024

test parameters:

@cmsbuild
Copy link
Contributor

cmsbuild commented Oct 4, 2024

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-46273/42096

@cmsbuild
Copy link
Contributor

cmsbuild commented Oct 4, 2024

A new Pull Request was created by @hqucms for master.

It involves the following packages:

  • DataFormats/BTauReco (reconstruction)
  • PhysicsTools/NanoAOD (xpog)
  • PhysicsTools/PatAlgos (reconstruction, xpog)
  • RecoBTag/FeatureTools (reconstruction)

@cmsbuild, @ftorrresd, @hqucms, @jfernan2, @mandrenguyen can you please review it and eventually sign? Thanks.
@AlexDeMoor, @AnnikaStein, @Ming-Yan, @Senphy, @ahinzmann, @andrzejnovak, @azotz, @castaned, @gkasieczka, @gouskos, @gpetruc, @hatakeyamak, @jdamgov, @jdolen, @mariadalfonso, @mbluj, @missirol, @mmarionncern, @mmusich, @nhanvtran, @rappoccio, @rovere, @schoef, @seemasharmafnal this is something you requested to watch as well.
@antoniovilela, @mandrenguyen, @rappoccio, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

@hqucms
Copy link
Contributor Author

hqucms commented Oct 4, 2024

please test

@cmsbuild
Copy link
Contributor

cmsbuild commented Oct 5, 2024

+1

Size: This PR adds an extra 60KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2b1f8f/41986/summary.html
COMMIT: bbdff12
CMSSW: CMSSW_14_2_X_2024-10-04-2300/el8_amd64_gcc12
Additional Tests: NANO
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/46273/41986/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

  • You potentially added 17 lines to the logs
  • ROOTFileChecks: Some differences in event products or their sizes found
  • Reco comparison results: 57 differences found in the comparisons
  • DQMHistoTests: Total files compared: 44
  • DQMHistoTests: Total histograms compared: 3331336
  • DQMHistoTests: Total failures: 4116
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 3327200
  • DQMHistoTests: Total skipped: 20
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 63.935 KiB( 43 files compared)
  • DQMHistoSizes: changed ( 11634.0,... ): 4.928 KiB Physics/NanoAODDQM
  • DQMHistoSizes: changed ( 13234.0,... ): 3.271 KiB Physics/NanoAODDQM
  • Checked 193 log files, 163 edm output root files, 44 DQM output files
  • TriggerResults: no differences found

NANO Comparison Summary

Summary:

  • You potentially added 932 lines to the logs
  • ROOTFileChecks: Some differences in event products or their sizes found
  • Reco comparison results: 0 differences found in the comparisons
  • DQMHistoTests: Total files compared: 21
  • DQMHistoTests: Total histograms compared: 55028
  • DQMHistoTests: Total failures: 0
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 55028
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 47.537 KiB( 20 files compared)
  • DQMHistoSizes: changed ( 2500.001,... ): 4.928 KiB Physics/NanoAODDQM
  • DQMHistoSizes: changed ( 2500.011,... ): 3.271 KiB Physics/NanoAODDQM
  • Checked 102 log files, 58 edm output root files, 21 DQM output files
  • TriggerResults: no differences found

Nano size comparison Summary:

Sample kb/ev ref kb/ev diff kb/ev ev/s/thd ref ev/s/thd diff rate mem/thd ref mem/thd
2500.001 2.961 2.949 0.012 ( +0.4% ) 3.02 3.10 -2.6% 6.070 6.059
2500.002 3.072 3.059 0.013 ( +0.4% ) 2.70 2.79 -3.2% 6.433 6.414
2500.003 3.014 3.003 0.011 ( +0.4% ) 2.83 2.90 -2.5% 6.406 6.423
2500.011 1.550 1.534 0.016 ( +1.1% ) 4.59 4.78 -3.9% 2.444 2.143
2500.012 2.050 2.031 0.019 ( +1.0% ) 2.78 2.91 -4.6% 2.628 2.184
2500.013 1.891 1.873 0.018 ( +1.0% ) 3.90 4.16 -6.2% 2.542 2.215
2500.021 0.022 0.022 0.000 ( +0.0% ) 0.93 0.99 -5.7% 2.391 2.373
2500.022 0.022 0.022 0.000 ( +0.0% ) 0.90 0.92 -2.2% 2.388 2.365
2500.023 0.022 0.022 0.000 ( +0.0% ) 0.87 0.91 -4.4% 2.253 2.247
2500.024 0.022 0.022 0.000 ( +0.0% ) 0.68 0.71 -4.1% 2.478 2.385
2500.031 0.035 0.035 0.000 ( +0.0% ) 0.82 0.86 -4.6% 2.440 2.412
2500.032 0.036 0.036 0.000 ( +0.0% ) 0.83 0.87 -4.2% 2.418 2.400
2500.033 0.037 0.037 -0.000 ( -0.1% ) 0.73 0.79 -6.8% 2.495 2.442
2500.034 0.036 0.036 0.000 ( +0.0% ) 0.76 0.80 -5.1% 2.475 2.420
2500.101 2.746 2.732 0.015 ( +0.5% ) 8.21 9.00 -8.8% 6.357 6.357
2500.111 1.407 1.389 0.018 ( +1.3% ) 15.52 20.33 -23.7% 2.258 2.235
2500.112 1.825 1.806 0.019 ( +1.1% ) 12.74 15.08 -15.5% 2.332 1.893
2500.131 0.747 0.747 0.000 ( +0.0% ) 18.37 18.36 +0.0% 1.406 1.400
2500.201 2.569 2.557 0.012 ( +0.5% ) 6.94 7.67 -9.5% 5.628 5.649
2500.211 1.722 1.700 0.022 ( +1.3% ) 13.64 17.93 -24.0% 2.010 1.983
2500.212 2.116 2.096 0.020 ( +1.0% ) 11.26 13.68 -17.7% 2.105 2.082
2500.221 2.017 2.004 0.013 ( +0.6% ) 7.23 7.77 -6.9% 2.033 2.012
2500.222 3.294 3.281 0.013 ( +0.4% ) 6.95 7.68 -9.6% 2.093 2.066
2500.223 8.981 8.968 0.013 ( +0.1% ) 2.59 2.67 -3.1% 2.067 2.121
2500.224 5.828 5.815 0.013 ( +0.2% ) 0.60 0.61 -1.0% 2.261 2.606
2500.225 5.846 5.833 0.013 ( +0.2% ) 0.58 0.59 -1.1% 2.274 2.617
2500.226 3.056 3.044 0.013 ( +0.4% ) 7.02 7.59 -7.6% 2.553 2.531
2500.227 1.437 1.437 0.000 ( +0.0% ) 11.93 11.95 -0.2% 1.426 1.421
2500.231 1.424 1.404 0.020 ( +1.5% ) 11.48 14.20 -19.1% 2.201 2.175
2500.232 2.321 2.301 0.020 ( +0.9% ) 11.26 14.30 -21.3% 2.298 2.273
2500.233 4.768 4.748 0.020 ( +0.4% ) 4.00 4.32 -7.5% 2.312 2.295
2500.234 3.539 3.518 0.020 ( +0.6% ) 0.76 0.77 -1.3% 2.082 2.313
2500.235 3.551 3.530 0.021 ( +0.6% ) 0.74 0.75 -1.7% 2.105 2.326
2500.236 2.166 2.145 0.020 ( +1.0% ) 11.46 14.31 -20.0% 2.287 2.271
2500.237 1.016 1.016 0.000 ( +0.0% ) 17.34 16.94 +2.4% 1.301 1.471
2500.241 9.404 9.404 0.000 ( +0.0% ) 3.77 3.66 +3.0% 1.728 1.951
2500.242 10.331 10.331 0.000 ( +0.0% ) 0.88 0.90 -2.1% 1.736 1.741
2500.243 2.712 2.712 0.000 ( +0.0% ) 7.72 8.59 -10.1% 1.079 1.076
2500.244 485.976 485.976 0.000 ( +0.0% ) 0.57 0.57 -0.2% 1.679 1.693
2500.245 823.202 823.202 0.000 ( +0.0% ) 0.75 0.75 +0.0% 1.663 1.666
2500.901 1.777 1.777 0.000 ( +0.0% ) 21.51 21.53 -0.1% 1.843 1.843
2500.902 1.626 1.626 0.000 ( +0.0% ) 22.52 20.29 +11.0% 1.769 1.768
2500.911 13.995 13.995 0.000 ( +0.0% ) 3.64 3.46 +5.2% 1.097 1.100
2500.912 0.438 0.240 0.198 ( +82.7% ) 1.53 1.71 -10.8% 0.974 0.976
2500.913 0.110 0.110 0.000 ( +0.0% ) 1.10 1.13 -2.4% 0.982 0.981

@hqucms
Copy link
Contributor Author

hqucms commented Oct 14, 2024

@jlidrych Could you please check the implementations here and make sure everything is correct? Please also provide a comparison of the model output using this NanoAOD-based implementation vs the training framework to see if the results agree with each other.

@hqucms
Copy link
Contributor Author

hqucms commented Oct 21, 2024

@jlidrych Could you please check the implementations here and make sure everything is correct? Please also provide a comparison of the model output using this NanoAOD-based implementation vs the training framework to see if the results agree with each other.

ping @jlidrych @cms-sw/muon-pog-l2

@jlidrych
Copy link
Contributor

My apologies for slow reply.
Here are the validation plots.
Please let me know if you would like to see some additional plots.

@hqucms
Copy link
Contributor Author

hqucms commented Oct 28, 2024

My apologies for slow reply. Here are the validation plots. Please let me know if you would like to see some additional plots.

Thanks a lot @jlidrych ! Looks good to me.

@cmsbuild
Copy link
Contributor

Pull request #46273 was updated. @ftorrresd, @hqucms, @jfernan2, @mandrenguyen can you please check and sign again.

@cmsbuild
Copy link
Contributor

-1

Failed Tests: RelVals-INPUT
Size: This PR adds an extra 40KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2b1f8f/42449/summary.html
COMMIT: 8a13edd
CMSSW: CMSSW_14_2_X_2024-10-29-1100/el8_amd64_gcc12
Additional Tests: NANO
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/46273/42449/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2b1f8f/42449/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2b1f8f/42449/git-merge-result

  • DAS Queries: The DAS query tests failed, see the summary page for details.

RelVals-INPUT

  • 2024.0000012024.000001_RunJetMET02024D_10k/step1_dasquery.log
  • 2024.1000012024.100001_RunJetMET02024C_10k/step1_dasquery.log
  • 2024.1010012024.101001_RunZeroBias2024C_10k/step1_dasquery.log
Expand to see more relval errors ...
  • 2024.000001
  • 2024.001001
  • 2024.100001
  • 2024.101001

Comparison Summary

Summary:

  • You potentially added 11 lines to the logs
  • ROOTFileChecks: Some differences in event products or their sizes found
  • Reco comparison results: 13 differences found in the comparisons
  • DQMHistoTests: Total files compared: 46
  • DQMHistoTests: Total histograms compared: 3569372
  • DQMHistoTests: Total failures: 425
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 3568927
  • DQMHistoTests: Total skipped: 20
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 63.935 KiB( 45 files compared)
  • DQMHistoSizes: changed ( 11634.0,... ): 4.928 KiB Physics/NanoAODDQM
  • DQMHistoSizes: changed ( 13234.0,... ): 3.271 KiB Physics/NanoAODDQM
  • Checked 201 log files, 171 edm output root files, 46 DQM output files
  • TriggerResults: no differences found

NANO Comparison Summary

Summary:

  • You potentially removed 1133 lines from the logs
  • ROOTFileChecks: Some differences in event products or their sizes found
  • Reco comparison results: 0 differences found in the comparisons
  • DQMHistoTests: Total files compared: 21
  • DQMHistoTests: Total histograms compared: 55313
  • DQMHistoTests: Total failures: 0
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 55313
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 47.537 KiB( 20 files compared)
  • DQMHistoSizes: changed ( 2500.001,... ): 4.928 KiB Physics/NanoAODDQM
  • DQMHistoSizes: changed ( 2500.011,... ): 3.271 KiB Physics/NanoAODDQM
  • Checked 102 log files, 58 edm output root files, 21 DQM output files
  • TriggerResults: no differences found

Nano size comparison Summary:

Sample kb/ev ref kb/ev diff kb/ev ev/s/thd ref ev/s/thd diff rate mem/thd ref mem/thd
2500.001 3.024 3.012 0.012 ( +0.4% ) 2.81 2.81 -0.0% 2.369 2.339
2500.002 3.138 3.124 0.013 ( +0.4% ) 2.49 2.52 -1.0% 2.809 2.362
2500.003 3.077 3.065 0.012 ( +0.4% ) 2.62 2.63 -0.2% 2.784 2.372
2500.011 1.605 1.591 0.014 ( +0.9% ) 4.19 4.27 -1.9% 2.470 2.445
2500.012 2.131 2.113 0.019 ( +0.9% ) 2.58 2.65 -2.7% 2.664 2.256
2500.013 1.954 1.936 0.018 ( +0.9% ) 3.57 3.67 -2.7% 2.568 2.247
2500.021 0.022 0.022 0.000 ( +0.2% ) 0.84 0.85 -0.6% 2.419 2.421
2500.022 0.022 0.022 0.000 ( +0.0% ) 0.81 0.81 -0.0% 2.419 2.416
2500.023 0.022 0.022 0.000 ( +0.0% ) 0.81 0.83 -2.5% 2.280 2.279
2500.024 0.022 0.022 0.000 ( +0.0% ) 0.62 0.62 -0.7% 2.496 2.502
2500.031 0.035 0.035 0.000 ( +0.0% ) 0.76 0.75 +0.4% 2.489 2.490
2500.032 0.036 0.036 0.000 ( +0.0% ) 0.76 0.76 -0.0% 2.448 2.449
2500.033 0.037 0.037 0.000 ( +0.0% ) 0.70 0.69 +0.5% 2.523 2.520
2500.034 0.036 0.036 0.000 ( +0.0% ) 0.70 0.70 -0.5% 2.513 2.502
2500.101 2.779 2.766 0.013 ( +0.5% ) 6.49 6.87 -5.5% 2.604 2.586
2500.111 1.439 1.419 0.020 ( +1.4% ) 12.72 15.69 -18.9% 2.310 2.299
2500.112 1.857 1.838 0.019 ( +1.0% ) 10.97 12.71 -13.7% 2.385 2.376
2500.131 0.747 0.747 0.000 ( +0.0% ) 16.66 16.42 +1.4% 1.512 1.509
2500.201 2.608 2.595 0.013 ( +0.5% ) 5.45 5.73 -4.8% 2.169 2.152
2500.211 1.779 1.757 0.021 ( +1.2% ) 11.05 13.74 -19.6% 2.377 2.358
2500.212 2.173 2.152 0.020 ( +0.9% ) 9.51 11.35 -16.3% 2.463 2.444
2500.221 2.028 2.016 0.013 ( +0.6% ) 5.71 6.09 -6.2% 2.092 2.075
2500.222 3.404 3.392 0.013 ( +0.4% ) 5.26 5.71 -7.8% 2.177 2.155
2500.223 9.369 9.357 0.013 ( +0.1% ) 1.65 1.66 -0.7% 2.256 2.233
2500.224 6.053 6.040 0.013 ( +0.2% ) 0.56 0.54 +2.4% 2.239 2.173
2500.225 6.084 6.072 0.013 ( +0.2% ) 0.52 0.52 +0.9% 2.318 2.252
2500.226 3.098 3.086 0.013 ( +0.4% ) 5.51 5.83 -5.5% 2.177 2.148
2500.227 1.437 1.437 0.000 ( +0.0% ) 10.72 10.53 +1.8% 1.433 1.426
2500.231 1.449 1.429 0.020 ( +1.4% ) 9.34 11.33 -17.5% 2.282 2.255
2500.232 2.432 2.412 0.020 ( +0.8% ) 8.87 10.26 -13.5% 2.370 2.353
2500.233 4.924 4.904 0.020 ( +0.4% ) 2.43 2.46 -1.5% 2.430 2.412
2500.234 3.690 3.670 0.020 ( +0.6% ) 0.71 0.70 +0.7% 2.404 2.146
2500.235 3.710 3.689 0.021 ( +0.6% ) 0.69 0.67 +1.8% 2.467 2.197
2500.236 2.222 2.202 0.020 ( +0.9% ) 9.21 11.11 -17.1% 2.368 2.352
2500.237 1.016 1.016 0.000 ( +0.0% ) 15.68 15.28 +2.6% 1.470 1.442
2500.241 9.404 9.404 0.000 ( +0.0% ) 3.44 3.32 +3.6% 1.929 1.917
2500.242 10.331 10.331 0.000 ( +0.0% ) 0.81 0.77 +4.9% 1.711 1.701
2500.243 2.712 2.712 0.000 ( +0.0% ) 7.82 7.53 +3.8% 1.081 1.076
2500.244 485.976 485.976 0.000 ( +0.0% ) 0.51 0.50 +2.3% 1.675 1.678
2500.245 823.202 823.202 0.000 ( +0.0% ) 0.66 0.65 +0.9% 1.671 1.654
2500.901 1.777 1.777 0.000 ( +0.0% ) 19.47 18.95 +2.8% 1.422 1.413
2500.902 1.626 1.626 0.000 ( +0.0% ) 19.51 19.58 -0.4% 1.325 1.318
2500.911 13.995 13.995 0.000 ( +0.0% ) 3.79 2.89 +31.0% 1.095 1.088
2500.912 0.199 0.310 -0.112 ( -36.0% ) 1.04 1.28 -18.5% 0.980 0.976
2500.913 0.110 0.110 0.000 ( +0.0% ) 1.05 1.06 -0.2% 0.980 0.976

@hqucms
Copy link
Contributor Author

hqucms commented Oct 29, 2024

+1

@hqucms
Copy link
Contributor Author

hqucms commented Oct 29, 2024

ignore tests-rejected with ib-failure

The dasquery failures are also present in the IBs.

@hqucms hqucms changed the title Rework of #46153 -- [NanoAOD] Add ParticleNet-based Muon ID [NanoAOD] Add ParticleNet-based Muon ID (Rework of #46153) Oct 29, 2024
@jfernan2
Copy link
Contributor

+1

@cmsbuild
Copy link
Contributor

This pull request is fully signed and it will be integrated in one of the next master IBs (test failures were overridden). This pull request will now be reviewed by the release team before it's merged. @rappoccio, @sextonkennedy, @mandrenguyen, @antoniovilela (and backports should be raised in the release meeting by the corresponding L2)
Notice This PR was tested with additional Pull Request(s), please also merge them if necessary: cms-data/PhysicsTools-NanoAOD#18

@mandrenguyen
Copy link
Contributor

+1

@cmsbuild cmsbuild merged commit 55d7248 into cms-sw:master Oct 31, 2024
13 of 14 checks passed
@mandrenguyen
Copy link
Contributor

I'm seeing an error that I believe comes from this PR:

customising the process with nanoAOD_customizeCommon from PhysicsTools/NanoAOD/nano_cff Starting cmsRun step4_NANO.py ----- Begin Fatal Exception 31-Oct-2024 21:04:55 CET----------------------- An exception of category 'ConfigFileReadError' occurred while [0] Processing the python configuration file named step4_NANO.py Exception Message: unknown python problem occurred. RuntimeError: An exception of category 'FileInPathError' occurred. Exception Message: edm::FileInPath unable to find file PhysicsTools/NanoAOD/data/PNetMuonId/model.onnx anywhere in the search path. The search path is defined by: CMSSW_SEARCH_PATH ${CMSSW_SEARCH_PATH} is: /afs/cern.ch/work/m/mnguyen/public/integration/CMSSW_14_2_X_2024-10-30-2300/poison:/afs/cern.ch/work/m/mnguyen/public/integration/CMSSW_14_2_X_2024-10-30-2300/src:/afs/cern.ch/work/m/mnguyen/public/integration/CMSSW_14_2_X_2024-10-30-2300/ext\ ernal/el8_amd64_gcc12/data:/cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-10-30-2300/src:/cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_2_X_2024-10-30-2300/external/el8_amd64_gcc12/data Current directory is: /tmp/mnguyen/135.4_ZEEFS_13

Can you take a look?

@mandrenguyen
Copy link
Contributor

^^ I think this was due to a delay in the 11AM IB becoming available such that the external cms-data was not found. I think you can disregard. Sorry for the noise.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants