Updating the WP of the Muon MVA ID #37250

andrea21z · 2022-03-15T12:30:19Z

PR description:

We have updated the Muon MVA ID model and we need to change the cut of the WP Tight, Cuts in IP are also added to the Tight working point.

Related PR.
This MVA has been presented in several Muon POG meetings: https://indico.cern.ch/event/1135971/contributions/4766180/attachments/2402974/4109990/muonMVAupdate_7mar2022.pdf
The analysis is documented in this Analysis Note: https://gitlab.cern.ch/tdr/notes/AN-21-107
Twiki: https://twiki.cern.ch/twiki/bin/view/CMS/LeptonMVAID
Cadiline: https://cms.cern.ch/iCMS/analysisadmin/cadilines?line=MUO-22-001
Related PR: to include the MVA model in data repository, in .onnx format -> Updating Muon MVA ID model cms-data/RecoMuon-MuonIdentification#8

PR validation:

We execute the basic tests suggested in the CMSSW PR instructions

if this PR is a backport please specify the original PR and why you need to backport that PR:

cmsbuild · 2022-03-15T12:38:11Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-37250/28852

This PR adds an extra 24KB to repository

cmsbuild · 2022-03-15T12:38:35Z

A new Pull Request was created by @andrea21z for master.

It involves the following packages:

PhysicsTools/PatAlgos (reconstruction)

@jpata, @cmsbuild, @clacaputo, @slava77 can you please review it and eventually sign? Thanks.
@AlexDeMoor, @rappoccio, @gouskos, @jdolen, @JyothsnaKomaragiri, @ahinzmann, @schoef, @emilbols, @jdamgov, @mbluj, @nhanvtran, @gkasieczka, @hatakeyamak, @gpetruc, @azotz, @mariadalfonso, @demuller, @andrzejnovak, @seemasharmafnal, @mmarionncern this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

jpata · 2022-03-15T13:16:04Z

This is a backport of the already merged #36179.

Probably there is some confusion about what a backport means. Please see here: https://cms-sw.github.io/tutorial-resolve-conflicts.html -> "Backporting a PR" for an explanation and clarify in the description if a backport is intended (in any case, a PR against master such as this one is needed).

jpata · 2022-03-15T13:16:20Z

test parameters:

pull_requests = Updating Muon MVA ID model cms-data/RecoMuon-MuonIdentification#8

jpata · 2022-03-15T13:16:33Z

@cmsbuild please test

andrea21z · 2022-03-15T18:00:11Z

"Backporting a PR" for an explanation and clarify in the description if a backport is intended (in any case, a PR against master such as this one is needed).

Ok, understood.

On the other hand, we just found a possible bug and we want to do a couple more checks to be sure before the PR is merged, so I think it's best to stop the PR checks for the moment. @jpata sorry for this ...

cmsbuild · 2022-03-15T18:03:47Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-33d550/23130/summary.html
COMMIT: 9bc993d
CMSSW: CMSSW_12_4_X_2022-03-15-1100/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/37250/23130/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

@mmusich HLTDeDxFilter: prevent variable length array bound evaluates to non-positive value #37229

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-33d550/23130/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-33d550/23130/git-merge-result

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 44 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3695377
DQMHistoTests: Total failures: 2
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3695353
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
Checked 204 log files, 45 edm output root files, 49 DQM output files
TriggerResults: no differences found

clacaputo · 2022-03-21T11:13:23Z

In the output there are 3 changes: a new branch with the output of the MVA (muon.mvaIDValue()), two new Selectors with the definition of the proposed Working Points (reco::Muon::MvaIDwpMedium, reco::Muon::MvaIDwpTight)

Hi @andrea21z , I don't see these changes in this PR, I suppose it is just a copy&paste from the description of #36179. Could you please edit the description, it's misleading. Thank you

andrea21z · 2022-03-21T15:30:52Z

In the output there are 3 changes: a new branch with the output of the MVA (muon.mvaIDValue()), two new Selectors with the definition of the proposed Working Points (reco::Muon::MvaIDwpMedium, reco::Muon::MvaIDwpTight)

Hi @andrea21z , I don't see these changes in this PR, I suppose it is just a copy&paste from the description of #36179. Could you please edit the description, it's misleading. Thank you

yes, sorry.

…uonMvaID_update

andrea21z · 2022-03-21T17:36:40Z

"Backporting a PR" for an explanation and clarify in the description if a backport is intended (in any case, a PR against master such as this one is needed).

Ok, understood.

On the other hand, we just found a possible bug and we want to do a couple more checks to be sure before the PR is merged, so I think it's best to stop the PR checks for the moment. @jpata sorry for this ...

@jpata @clacaputo you can continue with the review, all changes implemented.

clacaputo · 2022-03-23T17:36:25Z

Hello @andrea21z , do you have any plots comparing the old model and the new one?

andrea21z · 2022-03-24T10:44:42Z

Hello @andrea21z , do you have any plots comparing the old model and the new one?

Hello @clacaputo, what types of plots would you like to see? The only change with respect to the old model is that we removed a cut in Isolation used to select the muons to train the model.

clacaputo · 2022-03-24T17:36:51Z

Hello @andrea21z , do you have any plots comparing the old model and the new one?

Hello @clacaputo, what types of plots would you like to see? The only change with respect to the old model is that we removed a cut in Isolation used to select the muons to train the model.

Hi @andrea21z , a comparison of the output distribution mvaIDValue of the two models, so I can crosscheck that what I see in the comparisons are similar to what you have obtained. Thanks

clacaputo · 2022-03-28T11:21:41Z

Hi @andrea21z also a reference to a more recent presentation with the new model should be enough. Thanks

andrea21z · 2022-03-29T16:50:48Z

Hello @andrea21z , do you have any plots comparing the old model and the new one?

Hello @clacaputo, what types of plots would you like to see? The only change with respect to the old model is that we removed a cut in Isolation used to select the muons to train the model.

Hi @andrea21z , a comparison of the output distribution mvaIDValue of the two models, so I can crosscheck that what I see in the comparisons are similar to what you have obtained. Thanks

New MVA on the right and old MVA on the left.

andrea21z · 2022-03-29T16:52:57Z

Hi @andrea21z also a reference to a more recent presentation with the new model should be enough. Thanks

Here you can find the latest presentation in the Muon POG meeting,

clacaputo · 2022-03-30T15:19:24Z

Hi @andrea21z , thanks for the link. Could you please confirm me that the new model produces an output more shifted towards 0.9 for prompt muons wrt the old model? I can't get this info from the slides.

Here an examples for wf 1330.0 ZMM_13TeV_TuneCUETP8M1 run2 condition

andrea21z · 2022-03-30T15:51:17Z

Hi @andrea21z , thanks for the link. Could you please confirm me that the new model produces an output more shifted towards 0.9 for prompt muons wrt the old model? I can't get this info from the slides.

Here an examples for wf 1330.0 ZMM_13TeV_TuneCUETP8M1 run2 condition

Yes, I confirm this is expected. The output distribution of the new model (slide 14) has the maximum around 0.9 and prompt muons have values between 0.7 and 0.9.

clacaputo · 2022-03-30T21:41:06Z

+reconstruction

the PR changes muon MVA ID cut due to an updated MVA model Updating Muon MVA ID model cms-data/RecoMuon-MuonIdentification#8
reco changes are in line with the expectations
to be merged together with Updating Muon MVA ID model cms-data/RecoMuon-MuonIdentification#8

cmsbuild · 2022-03-30T21:41:38Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

perrotta · 2022-03-31T10:34:37Z

@cmsbuild please test
(tests have disappeared, and must be refreshed)

cmsbuild · 2022-03-31T15:27:56Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-33d550/23565/summary.html
COMMIT: 8db79f1
CMSSW: CMSSW_12_4_X_2022-03-30-2300/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/37250/23565/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

@bsunanda Run3-gex124E Try to remove compilation warnings in RecoParticleFlow/PFTracking #37369
@rovere Add TICL from Reco customise #37343
@bsunanda Run3-gex124C Try to avoid compilation warnings in L1Trigger/L1CaloTrigger #37353
@bsunanda Phase2-hgx303 Rearrange some of the parameters in the proper class in view of V17 geometry #37374

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-33d550/23565/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-33d550/23565/git-merge-result

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 45 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3591311
DQMHistoTests: Total failures: 8
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3591281
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
Checked 200 log files, 45 edm output root files, 48 DQM output files
TriggerResults: no differences found

perrotta · 2022-03-31T15:38:47Z

+1

Updating the WP of the Muon MVA ID

9bc993d

cmsbuild added this to the CMSSW_12_4_X milestone Mar 15, 2022

cmsbuild added code-checks-pending orp-pending pending-signatures reconstruction-pending tests-pending labels Mar 15, 2022

cmsbuild added code-checks-approved and removed code-checks-pending labels Mar 15, 2022

cmsbuild added the requires-external label Mar 15, 2022

cmsbuild added tests-started and removed tests-pending labels Mar 15, 2022

cmsbuild added tests-approved and removed tests-started labels Mar 15, 2022

andrea21z added 2 commits March 21, 2022 17:13

Merge branch 'CMSSW_12_4_X' of https://github.com/cms-sw/cmssw into M…

448eeb3

…uonMvaID_update

Adding IP cuts to Tight Muon ID

cdb0ac8

cmsbuild added code-checks-pending tests-pending and removed tests-approved code-checks-approved labels Mar 21, 2022

cmsbuild added tests-approved and removed tests-started labels Mar 22, 2022

cmsbuild added fully-signed reconstruction-approved and removed reconstruction-pending pending-signatures labels Mar 30, 2022

clacaputo mentioned this pull request Mar 30, 2022

Updating Muon MVA ID model cms-data/RecoMuon-MuonIdentification#8

Merged

cmsbuild added tests-started and removed tests-approved labels Mar 31, 2022

cmsbuild added tests-approved and removed tests-started labels Mar 31, 2022

cmsbuild added orp-approved and removed orp-pending labels Mar 31, 2022

cmsbuild merged commit 10f79a9 into cms-sw:master Mar 31, 2022

smuzaffar mentioned this pull request Mar 31, 2022

Update tag for RecoMuon-MuonIdentification to V01-14-00 cms-sw/cmsdist#7737

Merged

andrea21z mentioned this pull request Apr 21, 2022

Updating Muon MVA ID working point tight (Backport) #37647

Closed

cramonal mentioned this pull request Nov 8, 2022

Adding MVA ID to nanoAOD code #39355

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Updating the WP of the Muon MVA ID #37250

Updating the WP of the Muon MVA ID #37250

andrea21z commented Mar 15, 2022 •

edited

Loading

cmsbuild commented Mar 15, 2022

cmsbuild commented Mar 15, 2022

jpata commented Mar 15, 2022

jpata commented Mar 15, 2022

jpata commented Mar 15, 2022

andrea21z commented Mar 15, 2022

cmsbuild commented Mar 15, 2022

clacaputo commented Mar 21, 2022

andrea21z commented Mar 21, 2022

andrea21z commented Mar 21, 2022

clacaputo commented Mar 23, 2022

andrea21z commented Mar 24, 2022

clacaputo commented Mar 24, 2022

clacaputo commented Mar 28, 2022

andrea21z commented Mar 29, 2022

andrea21z commented Mar 29, 2022

clacaputo commented Mar 30, 2022

andrea21z commented Mar 30, 2022

clacaputo commented Mar 30, 2022 •

edited

Loading

cmsbuild commented Mar 30, 2022

perrotta commented Mar 31, 2022

cmsbuild commented Mar 31, 2022

perrotta commented Mar 31, 2022

Updating the WP of the Muon MVA ID #37250

Updating the WP of the Muon MVA ID #37250

Conversation

andrea21z commented Mar 15, 2022 • edited Loading

PR description:

PR validation:

if this PR is a backport please specify the original PR and why you need to backport that PR:

cmsbuild commented Mar 15, 2022

cmsbuild commented Mar 15, 2022

jpata commented Mar 15, 2022

jpata commented Mar 15, 2022

jpata commented Mar 15, 2022

andrea21z commented Mar 15, 2022

cmsbuild commented Mar 15, 2022

Comparison Summary

clacaputo commented Mar 21, 2022

andrea21z commented Mar 21, 2022

andrea21z commented Mar 21, 2022

clacaputo commented Mar 23, 2022

andrea21z commented Mar 24, 2022

clacaputo commented Mar 24, 2022

clacaputo commented Mar 28, 2022

andrea21z commented Mar 29, 2022

andrea21z commented Mar 29, 2022

clacaputo commented Mar 30, 2022

andrea21z commented Mar 30, 2022

clacaputo commented Mar 30, 2022 • edited Loading

cmsbuild commented Mar 30, 2022

perrotta commented Mar 31, 2022

cmsbuild commented Mar 31, 2022

Comparison Summary

perrotta commented Mar 31, 2022

andrea21z commented Mar 15, 2022 •

edited

Loading

clacaputo commented Mar 30, 2022 •

edited

Loading