[BTV] PUPPI ValueMap-compatible Deep taggers info producers #40803

nurfikri89 · 2023-02-17T08:34:54Z

PR description:

The main goal of this PR is to modify several BTV Deep taggers info producers, which are used to retrieve the features of each jet , to retrieve the puppi weight for each PF candidate from a ValueMap<float> produced by PuppiProducer. The (fallback) puppi weight value is set to be 1.0 before accessing the weight from the ValueMap. If users do not set the fallback_puppi_weight flag to be true and does not specify a source for puppi_value_map, an exception is thrown. A new optional flag is_weighted_jet is introduced so that users can make specify to apply the puppi weights on the PF constituents' four-vector. The flag should be set to true for Puppi jets.

A new function, setupPackedPuppi(), is defined in PhysicsTools/PatAlgos/python/tools/jetTools.py that can be called to setup a PuppiProducer instance and provides a common puppi weight ValueMap for any module which requires one.

PR validation:

Validation plots comparing the input variables before and after the fix can be found in this JMAR meeting virtual contribution.
passes the usual runTheMatrix test: runTheMatrix.py -l limited -i all --ibeos
passes JMENanoAOD workflows: runTheMatrix.py -i all --ibeos -l 10224.15,11024.15,25202.15,11634.15
passes reMiniAOD and reNanoAOD workflows: runTheMatrix.py -i all --ibeos -l 1325.518,2500.312

… weighted jets (e.g puppi jets)

…-tagging setup

…t() function if ValueMap not specified and explicitly asked to use fallback value.

…n when jet is weighted but no ValueMap is set. code-checks formatting.

nurfikri89 · 2023-02-17T08:36:33Z

FYI @AnnikaStein, @AlexDeMoor and @demuller

cmsbuild · 2023-02-17T08:40:45Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-40803/34234

This PR adds an extra 52KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File RecoBTag/FeatureTools/plugins/DeepBoostedJetTagInfoProducer.cc modified in PR(s): Custom Run 3 PFScouting NanoAOD #40438

cmsbuild · 2023-02-17T08:41:08Z

A new Pull Request was created by @nurfikri89 (Nurfikri Norjoharuddeen) for master.

It involves the following packages:

PhysicsTools/PatAlgos (xpog, reconstruction)
RecoBTag/FeatureTools (reconstruction)

@cmsbuild, @mandrenguyen, @clacaputo, @swertz, @vlimant can you please review it and eventually sign? Thanks.
@rappoccio, @gouskos, @hatakeyamak, @emilbols, @mbluj, @demuller, @seemasharmafnal, @mmarionncern, @missirol, @ahinzmann, @jdolen, @azotz, @hqucms, @jdamgov, @nhanvtran, @gkasieczka, @schoef, @andrzejnovak, @AlexDeMoor, @AnnikaStein, @JyothsnaKomaragiri, @gpetruc, @mariadalfonso this is something you requested to watch as well.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.

cms-bot commands are listed here

AnnikaStein · 2023-02-17T09:15:11Z

Hi @nurfikri89
as discussed via email, I support the modifications as they

are first of all necessary to actually use the weights from PuppiProducer for a potential re-NANO, in the packed_cand case
and second as this forces users to be explicit about what kind of weights they want to use.

Would we need a backport to 130X, given that one goal for this release reads „Possible re-miniAOD in EOY 2022“?

Best regards and thanks,
Annika

swertz · 2023-02-17T09:42:03Z

enable nano

swertz · 2023-02-17T09:42:09Z

please test

cmsbuild · 2023-02-17T14:13:47Z

-1

Failed Tests: RelVals-INPUT
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-8ee623/30698/summary.html
COMMIT: 79f3a5b
CMSSW: CMSSW_13_1_X_2023-02-16-2300/el8_amd64_gcc11
Additional Tests: NANO
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/40803/30698/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals-INPUT

The relvals timed out after 4 hours.

Comparison Summary

Summary:

You potentially added 5 lines to the logs
Reco comparison results: 164 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3556272
DQMHistoTests: Total failures: 20
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3556230
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
Checked 213 log files, 164 edm output root files, 49 DQM output files
TriggerResults: no differences found

NANO Comparison Summary

Summary:

You potentially added 3 lines to the logs
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 11
DQMHistoTests: Total histograms compared: 10829
DQMHistoTests: Total failures: 0
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 10829
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 10 files compared)
Checked 23 log files, 10 edm output root files, 11 DQM output files

Nano size comparison Summary:

Sample	kb/ev	ref kb/ev	diff kb/ev	ev/s/thd	ref ev/s/thd	diff rate	mem/thd	ref mem/thd
2500.31	2.232	2.232	0.000 ( +0.0% )	9.33	9.55	-2.2%	1.460	1.465
2500.311	2.323	2.323	0.000 ( +0.0% )	9.04	9.07	-0.3%	1.839	1.836
2500.312	2.277	2.277	0.000 ( +0.0% )	9.18	9.31	-1.5%	1.817	1.825
2500.33	1.100	1.100	0.000 ( +0.0% )	21.14	21.96	-3.8%	1.648	1.638
2500.331	1.394	1.394	0.000 ( +0.0% )	15.24	16.18	-5.8%	1.800	1.796
2500.332	1.326	1.326	0.000 ( +0.0% )	17.13	17.87	-4.2%	1.865	1.851
2500.401	2.139	2.139	0.000 ( +0.0% )	10.37	10.30	+0.7%	1.142	1.154
2500.501	1.711	1.711	0.000 ( +0.0% )	16.66	16.64	+0.1%	1.062	1.064
2500.511	1.124	1.124	0.000 ( +0.0% )	30.37	30.90	-1.7%	1.316	1.307
2500.601	2.050	2.050	0.000 ( +0.0% )	12.62	12.40	+1.7%	1.134	1.139

swertz · 2023-02-17T15:08:37Z

please test

nurfikri89 · 2023-03-23T14:48:18Z

But, for DeepDoubleX, the fallback weight of 1 was also used, so I would expect to see changes now that you've fixed that... Also, the weights were previously not used in the feature calculator https://github.com/cms-sw/cmssw/pull/40803/files#diff-0719f859daff442b1afbbf6b419ea2b413559a30f6924abc3c879024acb1e9b5, which is used by both DeepJet and DeepDoubleX/DeepBoostedJet, so I would expect changes to all of them?

@swertz I have updated the (slides) plots with the AK8 jet taggers added together. You are right, the NN taggers do change also with the updates but with the exception of ParticleNet because the use_puppiP4 is set to false for it:

cmssw/RecoBTag/ONNXRuntime/python/pfParticleNet_cff.py

Line 12 in 2f899a2

use_puppiP4 = False

swertz · 2023-03-23T15:28:49Z

Thanks for the check! I suppose we just didn't see any changes in AK8 jets in the tests because of the lack of statistics... Let me just refresh the tests and then we should be good to go.

swertz · 2023-03-23T15:29:05Z

please test

cmsbuild · 2023-03-23T17:56:38Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-8ee623/31566/summary.html
COMMIT: 79f3a5b
CMSSW: CMSSW_13_1_X_2023-03-22-2300/el8_amd64_gcc11
Additional Tests: NANO
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/40803/31566/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 172 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3552750
DQMHistoTests: Total failures: 20
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3552708
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
Checked 213 log files, 164 edm output root files, 49 DQM output files
TriggerResults: no differences found

NANO Comparison Summary

Summary:

You potentially added 1 lines to the logs
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 11
DQMHistoTests: Total histograms compared: 10829
DQMHistoTests: Total failures: 0
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 10829
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 10 files compared)
Checked 23 log files, 10 edm output root files, 11 DQM output files

Nano size comparison Summary:

Sample	kb/ev	ref kb/ev	diff kb/ev	ev/s/thd	ref ev/s/thd	diff rate	mem/thd	ref mem/thd
2500.31	2.224	2.224	0.000 ( +0.0% )	9.49	9.74	-2.5%	1.448	1.497
2500.311	2.323	2.323	0.000 ( +0.0% )	9.15	9.30	-1.6%	1.826	1.875
2500.312	2.277	2.277	0.000 ( +0.0% )	8.71	9.43	-7.7%	1.815	1.864
2500.33	1.099	1.099	0.000 ( +0.0% )	21.23	21.91	-3.1%	1.636	1.631
2500.331	1.394	1.394	0.000 ( +0.0% )	15.41	16.08	-4.2%	1.776	1.787
2500.332	1.326	1.326	0.000 ( +0.0% )	17.44	17.98	-3.0%	1.840	1.828
2500.401	2.138	2.138	0.000 ( +0.0% )	10.51	10.52	-0.1%	1.149	1.191
2500.501	1.711	1.711	0.000 ( +0.0% )	16.65	16.73	-0.5%	1.069	1.106
2500.511	1.124	1.124	0.000 ( +0.0% )	31.06	31.10	-0.1%	1.321	1.364
2500.601	2.050	2.050	0.000 ( +0.0% )	12.48	12.56	-0.6%	1.141	1.174

swertz · 2023-03-24T08:07:48Z

+1

Indeed there are also small changes to AK8 discriminators in MINI, see e.g. https://cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisons/CMSSW_13_1_X_2023-03-22-2300+8ee623/56230/validateJR/136.8311_RunJetHT2017FreMINIAOD/all_PAT_step2/

mandrenguyen · 2023-03-27T14:54:40Z

+1

cmsbuild · 2023-03-27T14:55:08Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

perrotta · 2023-03-27T21:53:09Z

+1

swertz · 2023-03-28T08:07:45Z

We will need to backport this to 13_0 for NanoV12.

mandrenguyen · 2023-03-29T05:51:45Z

type btv

…iValueMapForMini [BTV] Backport of #40803 (PUPPI ValueMap-compatible Deep taggers info producers)

nurfikri89 added 8 commits February 12, 2023 07:17

Retrieve puppi weights from ValueMap even with MiniAOD input

9d1f439

Apply puppi weight to constituent's energy and pT. Add optionflag for…

c46da01

… weighted jets (e.g puppi jets)

Specify puppi_value_map source and is_weighted_jet flag for DeepTag b…

7f8e0f1

…-tagging setup

Use puppi weights from ValueMap for pack_cand. Fallback to puppiWeigh…

78a1bf0

…t() function if ValueMap not specified and explicitly asked to use fallback value.

Fix typo

5a998af

Remove fallback_puppi_weight_ option

59f674b

Provide packedpuppi. Simplify setup.

e07e033

Set fallback puppi weight to 1.0 for packed_cand also. throw exceptio…

79f3a5b

…n when jet is weighted but no ValueMap is set. code-checks formatting.

cmsbuild added this to the CMSSW_13_1_X milestone Feb 17, 2023

cmsbuild added reconstruction-pending pending-signatures tests-pending orp-pending code-checks-pending xpog-pending labels Feb 17, 2023

cmsbuild added code-checks-approved and removed code-checks-pending labels Feb 17, 2023

cmsbuild added tests-started and removed tests-pending labels Feb 17, 2023

cmsbuild added tests-rejected and removed tests-started labels Feb 17, 2023

cmsbuild removed the tests-rejected label Feb 17, 2023

cmsbuild added tests-started and removed tests-approved labels Mar 23, 2023

cmsbuild added tests-approved and removed tests-started labels Mar 23, 2023

cmsbuild added xpog-approved and removed xpog-pending labels Mar 24, 2023

cmsbuild mentioned this pull request Mar 25, 2023

add track algo and originalAlgo to PackedCandidates #41183

Merged

cmsbuild added reconstruction-approved fully-signed and removed reconstruction-pending pending-signatures labels Mar 27, 2023

cmsbuild added orp-approved and removed orp-pending labels Mar 27, 2023

cmsbuild merged commit ac8fc87 into cms-sw:master Mar 27, 2023

cmsbuild mentioned this pull request Mar 28, 2023

Herwig lhe matching fix #40939

Merged

nurfikri89 mentioned this pull request Mar 28, 2023

[BTV] Backport of #40803 (PUPPI ValueMap-compatible Deep taggers info producers) #41210

Merged

cmsbuild added the btv label Mar 29, 2023

cmsbuild added a commit that referenced this pull request Apr 3, 2023

Merge pull request #41210 from nurfikri89/portFrom131XTo130X_btv_pupp…

cde8885

…iValueMapForMini [BTV] Backport of #40803 (PUPPI ValueMap-compatible Deep taggers info producers)

AlexDeMoor mentioned this pull request Apr 5, 2023

Introduce (Robust)ParTAK4 jet tagger, DeepJet model update for Run 3, remove DeepCSV from nano #41275

Merged

nurfikri89 deleted the from1300pre4_btv_puppiValueMapForMini branch June 7, 2023 18:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BTV] PUPPI ValueMap-compatible Deep taggers info producers #40803

[BTV] PUPPI ValueMap-compatible Deep taggers info producers #40803

nurfikri89 commented Feb 17, 2023

nurfikri89 commented Feb 17, 2023 •

edited

Loading

cmsbuild commented Feb 17, 2023

cmsbuild commented Feb 17, 2023

AnnikaStein commented Feb 17, 2023

swertz commented Feb 17, 2023

swertz commented Feb 17, 2023

cmsbuild commented Feb 17, 2023

swertz commented Feb 17, 2023

nurfikri89 commented Mar 23, 2023 •

edited

Loading

swertz commented Mar 23, 2023

swertz commented Mar 23, 2023

cmsbuild commented Mar 23, 2023

swertz commented Mar 24, 2023

mandrenguyen commented Mar 27, 2023

cmsbuild commented Mar 27, 2023

perrotta commented Mar 27, 2023

swertz commented Mar 28, 2023

mandrenguyen commented Mar 29, 2023

[BTV] PUPPI ValueMap-compatible Deep taggers info producers #40803

[BTV] PUPPI ValueMap-compatible Deep taggers info producers #40803

Conversation

nurfikri89 commented Feb 17, 2023

PR validation:

nurfikri89 commented Feb 17, 2023 • edited Loading

cmsbuild commented Feb 17, 2023

cmsbuild commented Feb 17, 2023

AnnikaStein commented Feb 17, 2023

swertz commented Feb 17, 2023

swertz commented Feb 17, 2023

cmsbuild commented Feb 17, 2023

RelVals-INPUT

Comparison Summary

NANO Comparison Summary

swertz commented Feb 17, 2023

nurfikri89 commented Mar 23, 2023 • edited Loading

swertz commented Mar 23, 2023

swertz commented Mar 23, 2023

cmsbuild commented Mar 23, 2023

Comparison Summary

NANO Comparison Summary

swertz commented Mar 24, 2023

mandrenguyen commented Mar 27, 2023

cmsbuild commented Mar 27, 2023

perrotta commented Mar 27, 2023

swertz commented Mar 28, 2023

mandrenguyen commented Mar 29, 2023

nurfikri89 commented Feb 17, 2023 •

edited

Loading

nurfikri89 commented Mar 23, 2023 •

edited

Loading