Pile-Up ID training and WPs for 2017 UL samples #30773

marinakolosova · 2020-07-17T08:05:37Z

PR description:

With this PR new training files and working points for UL 2017 samples are added. New pt bins are introduced and the code changed accordingly. (Studies shown here PU ID, UL17 slides)
There is also a PR for the new weights for PUJetID PR#11

PR validation:

Passed code-checks and code-format tests.
Passed runTheMatrix test workflows

FYI: @alefisico @camclean @singh-ramanpreet

cmsbuild · 2020-07-17T08:05:59Z

The code-checks are being triggered in jenkins.

cmsbuild · 2020-07-17T08:11:12Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-30773/17104

This PR adds an extra 28KB to repository

cmsbuild · 2020-07-17T08:11:34Z

A new Pull Request was created by @marinakolosova (Marina Kolosova) for master.

It involves the following packages:

RecoJets/JetProducers

@perrotta, @jpata, @cmsbuild, @slava77 can you please review it and eventually sign? Thanks.
@rappoccio, @jdolen, @ahinzmann, @jdamgov, @yslai, @nhanvtran, @gkasieczka, @clelange, @schoef, @mariadalfonso, @seemasharmafnal this is something you requested to watch as well.
@silviodonato, @dpiparo you are the release manager for this.

cms-bot commands are listed here

slava77 · 2020-07-17T12:43:24Z

@cmsbuild please test

cmsbuild · 2020-07-17T12:43:41Z

The tests are being triggered in jenkins.

CMSSW_11_2_X_2020-07-16-2300/slc7_amd64_gcc820: https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/8051/console Started: 2020/07/17 14:44

cmsbuild · 2020-07-17T13:37:00Z

-1

Tested at: ff17419

CMSSW: CMSSW_11_2_X_2020-07-16-2300
SCRAM_ARCH: slc7_amd64_gcc820
You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-328ad3/8051/summary.html

I found follow errors while testing this PR

Failed tests: RelVals

RelVals:

When I ran the RelVals I found an error in the following workflows:
136.7611 step2

runTheMatrix-results/136.7611_RunJetHT2016E_reminiaod+RunJetHT2016E_reminiaod+REMINIAOD_data2016_HIPM+HARVESTDR2_REMINIAOD_data2016_HIPM/step2_RunJetHT2016E_reminiaod+RunJetHT2016E_reminiaod+REMINIAOD_data2016_HIPM+HARVESTDR2_REMINIAOD_data2016_HIPM.log

136.8311 step2

runTheMatrix-results/136.8311_RunJetHT2017F_reminiaod+RunJetHT2017F_reminiaod+REMINIAOD_data2017+HARVEST2017_REMINIAOD_data2017/step2_RunJetHT2017F_reminiaod+RunJetHT2017F_reminiaod+REMINIAOD_data2017+HARVEST2017_REMINIAOD_data2017.log

136.88811 step2

runTheMatrix-results/136.88811_RunJetHT2018D_reminiaodUL+RunJetHT2018D_reminiaodUL+REMINIAOD_data2018UL+HARVEST2018_REMINIAOD_data2018UL/step2_RunJetHT2018D_reminiaodUL+RunJetHT2018D_reminiaodUL+REMINIAOD_data2018UL+HARVEST2018_REMINIAOD_data2018UL.log

140.56 step2

runTheMatrix-results/140.56_RunHI2018+RunHI2018+RECOHID18+HARVESTDHI18/step2_RunHI2018+RunHI2018+RECOHID18+HARVESTDHI18.log

4.53 step3

runTheMatrix-results/4.53_RunPhoton2012B+RunPhoton2012B+HLTD+RECODR1reHLT+HARVESTDR1reHLT/step3_RunPhoton2012B+RunPhoton2012B+HLTD+RECODR1reHLT+HARVESTDR1reHLT.log

136.731 step3

runTheMatrix-results/136.731_RunSinglePh2016B+RunSinglePh2016B+HLTDR2_2016+RECODR2_2016reHLT_skimSinglePh_HIPM+HARVESTDR2/step3_RunSinglePh2016B+RunSinglePh2016B+HLTDR2_2016+RECODR2_2016reHLT_skimSinglePh_HIPM+HARVESTDR2.log

136.793 step3

runTheMatrix-results/136.793_RunDoubleEG2017C+RunDoubleEG2017C+HLTDR2_2017+RECODR2_2017reHLT_skimDoubleEG_Prompt+HARVEST2017/step3_RunDoubleEG2017C+RunDoubleEG2017C+HLTDR2_2017+RECODR2_2017reHLT_skimDoubleEG_Prompt+HARVEST2017.log

136.874 step3

runTheMatrix-results/136.874_RunEGamma2018C+RunEGamma2018C+HLTDR2_2018+RECODR2_2018reHLT_skimEGamma_Offline_L1TEgDQM+HARVEST2018_L1TEgDQM/step3_RunEGamma2018C+RunEGamma2018C+HLTDR2_2018+RECODR2_2018reHLT_skimEGamma_Offline_L1TEgDQM+HARVEST2018_L1TEgDQM.log

158.01 step2

runTheMatrix-results/158.01_HydjetQ_reminiaodPbPb2018_INPUT+HydjetQ_reminiaodPbPb2018_INPUT+REMINIAODHI2018PPRECO+HARVESTHI2018PPRECOMINIAOD/step2_HydjetQ_reminiaodPbPb2018_INPUT+HydjetQ_reminiaodPbPb2018_INPUT+REMINIAODHI2018PPRECO+HARVESTHI2018PPRECOMINIAOD.log

135.4 step3

runTheMatrix-results/135.4_ZEE_13+ZEEFS_13+HARVESTUP15FS+MINIAODMCUP15FS/step3_ZEE_13+ZEEFS_13+HARVESTUP15FS+MINIAODMCUP15FS.log

1325.7 step2

runTheMatrix-results/1325.7_TTbar_13_94Xv2NanoAODINPUT+TTbar_13_94Xv2NanoAODINPUT+NANOEDMMC2017_94XMiniAODv2+HARVESTNANOAODMC2017_94XMiniAODv2/step2_TTbar_13_94Xv2NanoAODINPUT+TTbar_13_94Xv2NanoAODINPUT+NANOEDMMC2017_94XMiniAODv2+HARVESTNANOAODMC2017_94XMiniAODv2.log

1000.0 step2

runTheMatrix-results/1000.0_RunMinBias2011A+RunMinBias2011A+TIER0+SKIMD+HARVESTDfst2+ALCASPLIT/step2_RunMinBias2011A+RunMinBias2011A+TIER0+SKIMD+HARVESTDfst2+ALCASPLIT.log

1001.0 step2

runTheMatrix-results/1001.0_RunMinBias2011A+RunMinBias2011A+TIER0EXP+ALCAEXP+ALCAHARVDSIPIXELCALRUN1+ALCAHARVD1+ALCAHARVD2+ALCAHARVD3+ALCAHARVD4+ALCAHARVD5/step2_RunMinBias2011A+RunMinBias2011A+TIER0EXP+ALCAEXP+ALCAHARVDSIPIXELCALRUN1+ALCAHARVD1+ALCAHARVD2+ALCAHARVD3+ALCAHARVD4+ALCAHARVD5.log

1306.0 step3

runTheMatrix-results/1306.0_SingleMuPt1_UP15+SingleMuPt1_UP15+DIGIUP15+RECOUP15+HARVESTUP15/step3_SingleMuPt1_UP15+SingleMuPt1_UP15+DIGIUP15+RECOUP15+HARVESTUP15.log

9.0 step3

runTheMatrix-results/9.0_Higgs200ChargedTaus+Higgs200ChargedTaus+DIGI+RECO+HARVEST/step3_Higgs200ChargedTaus+Higgs200ChargedTaus+DIGI+RECO+HARVEST.log

1330.0 step3

runTheMatrix-results/1330.0_ZMM_13+ZMM_13+DIGIUP15+RECOUP15_L1TMuDQM+HARVESTUP15_L1TMuDQM+NANOUP15/step3_ZMM_13+ZMM_13+DIGIUP15+RECOUP15_L1TMuDQM+HARVESTUP15_L1TMuDQM+NANOUP15.log

25.0 step3

runTheMatrix-results/25.0_TTbar+TTbar+DIGI+RECOAlCaCalo+HARVEST+ALCATT/step3_TTbar+TTbar+DIGI+RECOAlCaCalo+HARVEST+ALCATT.log

23234.1001 step1

runTheMatrix-results/23234.1001_TTbar_14TeV+RecoFullGlobal_TestOldDigi_2026D49+HARVESTFullGlobal_2026D49/step1_TTbar_14TeV+RecoFullGlobal_TestOldDigi_2026D49+HARVESTFullGlobal_2026D49.log

158.0 step3

runTheMatrix-results/158.0_HydjetQ_B12_5020GeV_2018_ppReco+HydjetQ_B12_5020GeV_2018_ppReco+DIGIHI2018PPRECO+RECOHI2018PPRECO+ALCARECOHI2018PPRECO+HARVESTHI2018PPRECO/step3_HydjetQ_B12_5020GeV_2018_ppReco+HydjetQ_B12_5020GeV_2018_ppReco+DIGIHI2018PPRECO+RECOHI2018PPRECO+ALCARECOHI2018PPRECO+HARVESTHI2018PPRECO.log

10042.0 step3

runTheMatrix-results/10042.0_ZMM_13+ZMM_13TeV_TuneCUETP8M1_2017_GenSimFull+DigiFull_2017+RecoFull_2017+HARVESTFull_2017+ALCAFull_2017+NanoFull_2017/step3_ZMM_13+ZMM_13TeV_TuneCUETP8M1_2017_GenSimFull+DigiFull_2017+RecoFull_2017+HARVESTFull_2017+ALCAFull_2017+NanoFull_2017.log

10024.0 step3

runTheMatrix-results/10024.0_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2017_GenSimFull+DigiFull_2017+RecoFull_2017+HARVESTFull_2017+ALCAFull_2017+NanoFull_2017/step3_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2017_GenSimFull+DigiFull_2017+RecoFull_2017+HARVESTFull_2017+ALCAFull_2017+NanoFull_2017.log

10824.0 step3

runTheMatrix-results/10824.0_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2018_GenSimFull+DigiFull_2018+RecoFull_2018+HARVESTFull_2018+ALCAFull_2018+NanoFull_2018/step3_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2018_GenSimFull+DigiFull_2018+RecoFull_2018+HARVESTFull_2018+ALCAFull_2018+NanoFull_2018.log

11634.0 step3

runTheMatrix-results/11634.0_TTbar_14TeV+TTbar_14TeV_TuneCP5_2021_GenSimFull+DigiFull_2021+RecoFull_2021+HARVESTFull_2021+ALCAFull_2021/step3_TTbar_14TeV+TTbar_14TeV_TuneCP5_2021_GenSimFull+DigiFull_2021+RecoFull_2021+HARVESTFull_2021+ALCAFull_2021.log

25202.0 step3

runTheMatrix-results/25202.0_TTbar_13+TTbar_13+DIGIUP15_PU25+RECOUP15_PU25+HARVESTUP15_PU25+NANOUP15_PU25/step3_TTbar_13+TTbar_13+DIGIUP15_PU25+RECOUP15_PU25+HARVESTUP15_PU25+NANOUP15_PU25.log

12434.0 step3

runTheMatrix-results/12434.0_TTbar_14TeV+TTbar_14TeV_TuneCP5_2023_GenSimFull+DigiFull_2023+RecoFull_2023+HARVESTFull_2023+ALCAFull_2023/step3_TTbar_14TeV+TTbar_14TeV_TuneCP5_2023_GenSimFull+DigiFull_2023+RecoFull_2023+HARVESTFull_2023+ALCAFull_2023.log

10224.0 step3

runTheMatrix-results/10224.0_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2017PU_GenSimFull+DigiFullPU_2017PU+RecoFullPU_2017PU+HARVESTFullPU_2017PU+NanoFull_2017PU/step3_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2017PU_GenSimFull+DigiFullPU_2017PU+RecoFullPU_2017PU+HARVESTFullPU_2017PU+NanoFull_2017PU.log

23234.0 step3

runTheMatrix-results/23234.0_TTbar_14TeV+TTbar_14TeV_TuneCP5_2026D49_GenSimHLBeamSpotFull14+DigiFullTrigger_2026D49+RecoFullGlobal_2026D49+HARVESTFullGlobal_2026D49/step3_TTbar_14TeV+TTbar_14TeV_TuneCP5_2026D49_GenSimHLBeamSpotFull14+DigiFullTrigger_2026D49+RecoFullGlobal_2026D49+HARVESTFullGlobal_2026D49.log

28234.0 step3

runTheMatrix-results/28234.0_TTbar_14TeV+TTbar_14TeV_TuneCP5_2026D60_GenSimHLBeamSpotFull14+DigiFullTrigger_2026D60+RecoFullGlobal_2026D60+HARVESTFullGlobal_2026D60/step3_TTbar_14TeV+TTbar_14TeV_TuneCP5_2026D60_GenSimHLBeamSpotFull14+DigiFullTrigger_2026D60+RecoFullGlobal_2026D60+HARVESTFullGlobal_2026D60.log

250202.181 step4

runTheMatrix-results/250202.181_TTbar_13UP18+TTbar_13UP18+PREMIXUP18_PU25+DIGIPRMXLOCALUP18_PU25+RECOPRMXUP18_PU25+HARVESTUP18_PU25/step4_TTbar_13UP18+TTbar_13UP18+PREMIXUP18_PU25+DIGIPRMXLOCALUP18_PU25+RECOPRMXUP18_PU25+HARVESTUP18_PU25.log

cmsbuild · 2020-07-17T13:37:02Z

Comparison not run due to runTheMatrix errors (RelVals and Igprof tests were also skipped)

alefisico · 2020-07-17T14:36:50Z

Just to mention that these validation tests are going to fail because the PR needs these files: cms-data/RecoJets-JetProducers#11

slava77 · 2020-07-17T14:59:17Z

test parameters

pull_request = Including weights for UL17 PUjetID training cms-data/RecoJets-JetProducers#11

slava77 · 2020-07-17T14:59:29Z

@cmsbuild please test

cmsbuild · 2020-07-17T14:59:36Z

The tests are being triggered in jenkins.
Tested with other pull request(s) cms-data/RecoJets-JetProducers#11

CMSSW_11_2_X_2020-07-17-1100/slc7_amd64_gcc820: https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/8059/console Started: 2020/07/17 17:05

slava77 · 2020-07-17T15:09:10Z

RecoJets/JetProducers/python/PileupJetIDCutParams_cfi.py

@@ -46,19 +77,22 @@
    Pt010_Tight    = cms.vdouble( 0.26, -0.34, -0.24, -0.26),
    Pt1020_Tight   = cms.vdouble( 0.26, -0.34, -0.24, -0.26),
    Pt2030_Tight   = cms.vdouble( 0.26, -0.34, -0.24, -0.26),
-    Pt3050_Tight   = cms.vdouble( 0.62, -0.21, -0.07, -0.03),
+    Pt3040_Tight   = cms.vdouble( 0.62, -0.21, -0.07, -0.03),
+    Pt4050_Tight   = cms.vdouble( 0.62, -0.21, -0.07, -0.03),


are these WPs actually used?
I'm curious when is the opportunity to clean up this file.

sorry you mean the 80X WPs or in general the Pt3040_* and Pt4050_*?
If it is the Pt** WPs, it is because we split the Pt3050 WPs in UL campaign.

@slava77 the 80X WPs are indeed obsolete.

I meant 80X, 53X and alike.

However, we need this PR ASAP. We will make sure to clean up this file on the next iteration.

cmsbuild · 2020-07-17T16:41:41Z

+1
Tested at: ff17419
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-80825c/8059/summary.html
CMSSW: CMSSW_11_2_X_2020-07-17-1100
SCRAM_ARCH: slc7_amd64_gcc820

cmsbuild · 2020-07-17T16:41:47Z

Comparison job queued.

cmsbuild · 2020-07-17T18:21:55Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-80825c/8059/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 45 differences found in the comparisons
DQMHistoTests: Total files compared: 34
DQMHistoTests: Total histograms compared: 2525996
DQMHistoTests: Total failures: 854
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 2525094
DQMHistoTests: Total skipped: 47
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.004 KiB( 33 files compared)
DQMHistoSizes: changed ( 10224.0 ): 0.004 KiB MessageLogger/Warnings
Checked 144 log files, 17 edm output root files, 34 DQM output files

slava77 · 2020-07-20T22:24:15Z

+1

for #30773 ff17419

code changes are in line with the PR description and linked slides (30-50 is split in two; the new IDs full_106x_UL17_chs are applied to all; the cuts in the slides are in agreement with changes in the code)
jenkins tests pass and comparisons with the baseline show differences in the MVAPUID variables

data shows more clearly a drop in the fraction of passing jets
e.g. wf 136.874 (jetHT 2018; red is this PR)

MC apparently has much less change, but the available tests done by the bot are done on just 10 events (compared to 100 for data).

These appear to be in line with the plots in the linked slides in the PR description.

cms-data/RecoJets-JetProducers#11 is needed when this PR is merged

cmsbuild · 2020-07-20T22:24:36Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo (and backports should be raised in the release meeting by the corresponding L2)

silviodonato · 2020-07-21T08:10:08Z

+1
cms-data/RecoJets-JetProducers#11 and
cms-sw/cmsdist#6104
are already merged

marinakolosova added 2 commits July 9, 2020 07:32

[PUID]: updated pt bins and added 2017 UL WPs

e9d7e90

[PUID]: passed code-checks

ff17419

cmsbuild added this to the CMSSW_11_2_X milestone Jul 17, 2020

cmsbuild added code-checks-pending comparison-pending orp-pending pending-signatures reconstruction-pending tests-pending labels Jul 17, 2020

cmsbuild added code-checks-approved and removed code-checks-pending labels Jul 17, 2020

cmsbuild added tests-started and removed tests-pending labels Jul 17, 2020

cmsbuild added comparison-notrun tests-rejected and removed comparison-pending tests-started labels Jul 17, 2020

cmsbuild added comparison-pending and removed comparison-notrun tests-rejected labels Jul 17, 2020

cmsbuild added the tests-pending label Jul 17, 2020

cmsbuild mentioned this pull request Jul 17, 2020

Including weights for UL17 PUjetID training cms-data/RecoJets-JetProducers#11

Merged

cmsbuild added requires-external tests-started and removed tests-pending labels Jul 17, 2020

slava77 reviewed Jul 17, 2020

View reviewed changes

slava77 mentioned this pull request Jul 17, 2020

cleanup RecoJets/JetProducers/python/PileupJetIDCutParams_cfi.py from old/unsupported PSets #30776

Closed

cmsbuild added tests-approved and removed tests-started labels Jul 17, 2020

cmsbuild added comparison-available and removed comparison-pending labels Jul 17, 2020

cmsbuild added fully-signed reconstruction-approved and removed pending-signatures reconstruction-pending labels Jul 20, 2020

cmsbuild added orp-approved and removed orp-pending labels Jul 21, 2020

cmsbuild merged commit 78301dc into cms-sw:master Jul 21, 2020

marinakolosova mentioned this pull request Jul 28, 2020

106X backport of: Pile-Up ID training and WPs for 2017 UL samples #30935

Merged

slava77 mentioned this pull request Dec 19, 2020

Pileup Id training for UL18 #32418

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pile-Up ID training and WPs for 2017 UL samples #30773

Pile-Up ID training and WPs for 2017 UL samples #30773

marinakolosova commented Jul 17, 2020

cmsbuild commented Jul 17, 2020

cmsbuild commented Jul 17, 2020

cmsbuild commented Jul 17, 2020

slava77 commented Jul 17, 2020

cmsbuild commented Jul 17, 2020 •

edited

Loading

cmsbuild commented Jul 17, 2020

cmsbuild commented Jul 17, 2020

alefisico commented Jul 17, 2020

slava77 commented Jul 17, 2020

slava77 commented Jul 17, 2020

cmsbuild commented Jul 17, 2020 •

edited

Loading

slava77 Jul 17, 2020

alefisico Jul 17, 2020 •

edited

Loading

camclean Jul 17, 2020

slava77 Jul 17, 2020

camclean Jul 17, 2020

cmsbuild commented Jul 17, 2020

cmsbuild commented Jul 17, 2020

cmsbuild commented Jul 17, 2020

slava77 commented Jul 20, 2020 •

edited

Loading

cmsbuild commented Jul 20, 2020

silviodonato commented Jul 21, 2020 •

edited

Loading

Pile-Up ID training and WPs for 2017 UL samples #30773

Pile-Up ID training and WPs for 2017 UL samples #30773

Conversation

marinakolosova commented Jul 17, 2020

PR description:

PR validation:

cmsbuild commented Jul 17, 2020

cmsbuild commented Jul 17, 2020

cmsbuild commented Jul 17, 2020

slava77 commented Jul 17, 2020

cmsbuild commented Jul 17, 2020 • edited Loading

cmsbuild commented Jul 17, 2020

cmsbuild commented Jul 17, 2020

alefisico commented Jul 17, 2020

slava77 commented Jul 17, 2020

slava77 commented Jul 17, 2020

cmsbuild commented Jul 17, 2020 • edited Loading

slava77 Jul 17, 2020

Choose a reason for hiding this comment

alefisico Jul 17, 2020 • edited Loading

Choose a reason for hiding this comment

camclean Jul 17, 2020

Choose a reason for hiding this comment

slava77 Jul 17, 2020

Choose a reason for hiding this comment

camclean Jul 17, 2020

Choose a reason for hiding this comment

cmsbuild commented Jul 17, 2020

cmsbuild commented Jul 17, 2020

cmsbuild commented Jul 17, 2020

slava77 commented Jul 20, 2020 • edited Loading

cmsbuild commented Jul 20, 2020

silviodonato commented Jul 21, 2020 • edited Loading

cmsbuild commented Jul 17, 2020 •

edited

Loading

cmsbuild commented Jul 17, 2020 •

edited

Loading

alefisico Jul 17, 2020 •

edited

Loading

slava77 commented Jul 20, 2020 •

edited

Loading

silviodonato commented Jul 21, 2020 •

edited

Loading