Nano: fix and additions to collection matching, remove cross-cleaning #40162

swertz · 2022-11-27T19:11:54Z

PR description:

Fill in a few gaps in the matching between collections in Nano:

boostedTaus were not used in the matching at all, resulting in the existing branch boostedTau_jetIdx to be always -1
add matching between secondary vertices and other objects (only store indices to SVs in the other objects, the SV table is not modified)
add the following matching cases (matching done by common PF cands) and corresponding index branches:
- jets with boostedTaus (no boostedTau index branch added to the jets, but boostedTau_jetIdx is now usable)
- jets with SVs (2 indices + nSV stored in jets)
- lowPtElectrons with photons (photon index stored in lowPtElectrons)
- taus with electrons (electron index stored in taus)
- taus with muons (muon index stored in taus)
- taus with SVs (2 indices + nSV stored in taus)
- electrons with SVs (SV index stored in electrons)
- muons with SVs (SV index stored in muons)

Also remove the object cleanmask branches because they weren't useful in practice as cross-cleaning is heavily analysis-dependent (depends on the object selection). The cleaning can always be done easily a posteriori using the cross-matching indices stored. The cross-cleaning module is kept in place in case people want to use it in private productions.

NanoAOD file size increase from these changes is 0.02 kb/event.

PR validation:

Ran with runTheMatrix.py --what nano --site "" -l 2500.601

swertz · 2022-11-27T19:14:17Z

enable nano

cmsbuild · 2022-11-27T19:27:23Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-40162/33167

This PR adds an extra 48KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File PhysicsTools/NanoAOD/python/nano_cff.py modified in PR(s): Refactoring gen weight storage in EDM + Nano integration #32167

Code check has found code style and quality issues which could be resolved by applying following patch(s)

code-format:
https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-40162/33167/code-format.patch
e.g. curl -k https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-40162/33167/code-format.patch | patch -p1
You can also run scram build code-format to apply code format directly

cmsbuild · 2022-11-27T20:25:22Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-40162/33168

This PR adds an extra 48KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File PhysicsTools/NanoAOD/python/nano_cff.py modified in PR(s): Refactoring gen weight storage in EDM + Nano integration #32167

cmsbuild · 2022-11-27T20:25:44Z

A new Pull Request was created by @swertz (Sébastien Wertz) for master.

It involves the following packages:

PhysicsTools/NanoAOD (xpog)
PhysicsTools/PatAlgos (xpog, reconstruction)

@cmsbuild, @mandrenguyen, @clacaputo, @swertz, @vlimant can you please review it and eventually sign? Thanks.
@AlexDeMoor, @rappoccio, @gouskos, @jdolen, @JyothsnaKomaragiri, @ahinzmann, @AnnikaStein, @schoef, @emilbols, @jdamgov, @mbluj, @nhanvtran, @gkasieczka, @hatakeyamak, @gpetruc, @azotz, @mariadalfonso, @demuller, @andrzejnovak, @seemasharmafnal, @mmarionncern this is something you requested to watch as well.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.

cms-bot commands are listed here

swertz · 2022-11-27T20:26:57Z

please test

cmsbuild · 2022-11-27T23:54:19Z

-1

Failed Tests: RelVals-INPUT RelVals-NANO
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-d141d2/29287/summary.html
COMMIT: b3e3694
CMSSW: CMSSW_13_0_X_2022-11-27-0000/el8_amd64_gcc11
Additional Tests: NANO
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/40162/29287/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals-INPUT

2500.52500.5_mc124Xrun3_v10+TTbarMINIAOD12.4+NANO_mc12.4_v10+HRV_NANO_mc/step2_mc124Xrun3_v10+TTbarMINIAOD12.4+NANO_mc12.4_v10+HRV_NANO_mc.log
2500.42500.4_mc122Xrun3_v10+TTbarMINIAOD12.2+NANO_mc12.2_v10+HRV_NANO_mc/step2_mc122Xrun3_v10+TTbarMINIAOD12.2+NANO_mc12.2_v10+HRV_NANO_mc.log
2500.512500.51_data124Xrun3_v10+MuonEG2022MINIAOD12.4+NANO_data12.4_v10+HRV_NANO_data/step2_data124Xrun3_v10+MuonEG2022MINIAOD12.4+NANO_data12.4_v10+HRV_NANO_data.log

Expand to see more relval errors ...

RelVals-NANO

2500.62500.6_mc126X_v10+TTBarMINIAOD12.6+NANO_mc12.6_v10+HRV_NANO_mc/step2_mc126X_v10+TTBarMINIAOD12.6+NANO_mc12.6_v10+HRV_NANO_mc.log
2500.52500.5_mc124Xrun3_v10+TTbarMINIAOD12.4+NANO_mc12.4_v10+HRV_NANO_mc/step2_mc124Xrun3_v10+TTbarMINIAOD12.4+NANO_mc12.4_v10+HRV_NANO_mc.log
2500.42500.4_mc122Xrun3_v10+TTbarMINIAOD12.2+NANO_mc12.2_v10+HRV_NANO_mc/step2_mc122Xrun3_v10+TTbarMINIAOD12.2+NANO_mc12.2_v10+HRV_NANO_mc.log

Expand to see more relval errors ...

Comparison Summary

Summary:

No significant changes to the logs found
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 132 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3417239
DQMHistoTests: Total failures: 102
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3417115
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
Checked 206 log files, 48 edm output root files, 48 DQM output files
TriggerResults: no differences found

cmsbuild · 2022-11-28T09:22:37Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-40162/33170

This PR adds an extra 48KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File PhysicsTools/NanoAOD/python/nano_cff.py modified in PR(s): Refactoring gen weight storage in EDM + Nano integration #32167

swertz · 2022-12-03T19:19:23Z

For some reason it seems other commits have been included in the test? Obviously the commits in this PR, which were previously validated, have not changed. Would you rather running the tests again @perrotta ?

perrotta · 2022-12-03T19:53:55Z

Please test
(A new IB is now available)

cmsbuild · 2022-12-03T23:54:34Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-d141d2/29446/summary.html
COMMIT: cd0cf1b
CMSSW: CMSSW_13_0_X_2022-12-03-1100/el8_amd64_gcc11
Additional Tests: NANO
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/40162/29446/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 128 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3421214
DQMHistoTests: Total failures: 152
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3421040
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -6.326 KiB( 47 files compared)
DQMHistoSizes: changed ( 11634.0,... ): -0.774 KiB Physics/NanoAODDQM
DQMHistoSizes: changed ( 13234.0,... ): -0.454 KiB Physics/NanoAODDQM
Checked 206 log files, 158 edm output root files, 48 DQM output files
TriggerResults: no differences found

NANO Comparison Summary

Summary:

No significant changes to the logs found
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 69 differences found in the comparisons
DQMHistoTests: Total files compared: 15
DQMHistoTests: Total histograms compared: 15192
DQMHistoTests: Total failures: 0
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 15192
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -9.236 KiB( 14 files compared)
DQMHistoSizes: changed ( 2500.311,... ): -0.774 KiB Physics/NanoAODDQM
DQMHistoSizes: changed ( 2500.331,... ): -0.454 KiB Physics/NanoAODDQM
Checked 31 log files, 14 edm output root files, 15 DQM output files

Nano size comparison Summary:

Sample	kb/ev	ref kb/ev	diff kb/ev	ev/s/thd	ref ev/s/thd	diff rate	mem/thd	ref mem/thd
2500.31	2.200	2.190	0.010 ( +0.5% )	9.39	9.51	-1.3%	1.546	1.529
2500.311	2.322	2.312	0.010 ( +0.4% )	8.86	9.07	-2.4%	1.908	1.904
2500.312	2.276	2.264	0.012 ( +0.5% )	9.23	9.17	+0.6%	1.897	1.892
2500.33	1.089	1.083	0.006 ( +0.6% )	21.08	21.12	-0.2%	1.709	1.690
2500.331	1.388	1.379	0.009 ( +0.6% )	15.76	15.80	-0.2%	1.856	1.852
2500.332	1.319	1.312	0.007 ( +0.5% )	17.66	17.41	+1.5%	1.810	1.798
2500.4	2.097	2.097	0.000 ( +0.0% )	10.13	10.04	+0.9%	1.444	1.429
2500.401	2.146	2.135	0.010 ( +0.5% )	10.20	10.18	+0.2%	1.247	1.225
2500.5	1.668	1.668	0.000 ( +0.0% )	16.30	16.13	+1.1%	1.162	1.137
2500.501	1.711	1.705	0.006 ( +0.4% )	16.29	16.14	+0.9%	1.160	1.134
2500.51	1.071	1.071	0.000 ( +0.0% )	28.65	28.08	+2.0%	1.449	1.422
2500.511	1.120	1.113	0.007 ( +0.6% )	28.15	27.83	+1.1%	1.470	1.443
2500.6	2.006	2.006	0.000 ( +0.0% )	12.96	12.93	+0.2%	1.221	1.203
2500.601	2.057	2.045	0.011 ( +0.5% )	12.95	12.82	+1.0%	1.224	1.207

swertz · 2022-12-04T07:15:28Z

+1

(though there are again spurious differences that are entirely unrelated with this PR...)

mandrenguyen · 2022-12-05T15:51:43Z

+reconstruction
These spurious changes are a bit irritating, but looking at the code which is touched there is no way anything but nano is updated.

cmsbuild · 2022-12-05T15:52:08Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

rappoccio · 2022-12-06T14:16:23Z

ping bot

smuzaffar · 2022-12-06T14:19:33Z

@rappoccio i think orp signatures are still missing for this PR

rappoccio · 2022-12-06T14:25:26Z

@smuzaffar the Jenkins tests are still showing as "pending" and waiting for a test command, which was already done, so I was hoping it would come back to life :)

swertz · 2022-12-06T14:27:30Z

They are shown as pending for #40183, I think it got confused because I opened that one from the same branch...

smuzaffar · 2022-12-06T14:29:14Z

@rappoccio , this pending test bot/40183/jenkins is for PR #40183 ( which is close now). Note that the checks belong to the commit, as the same commit is used for #40183 and #40162 that is why you see all the checks in both PRs. As far as this PR is concerned, all checks are passed and you should be able to issue +1 to merge it

rappoccio · 2022-12-06T14:29:51Z

+1

Jenkins is confused because of the same branch being used for two PRs. (Let's avoid that in the future @swertz )

smuzaffar · 2022-12-06T14:36:57Z

Jenkins is confused because of the same branch being used for two PRs. (Let's avoid that in the future @swertz )

It is not jenkins but github feature, checks belong to commit instead of PR :-) I think there is no harm opening multiple PRs using same branch/commit. As labels belong to PRs, so test-approved label from bot is a good indicator that PR tests passed.

cmsbuild added this to the CMSSW_13_0_X milestone Nov 27, 2022

cmsbuild added code-checks-pending orp-pending pending-signatures reconstruction-pending tests-pending xpog-pending labels Nov 27, 2022

cmsbuild added code-checks-rejected code-checks-pending and removed code-checks-pending code-checks-rejected labels Nov 27, 2022

cmsbuild added code-checks-approved and removed code-checks-pending labels Nov 27, 2022

cmsbuild added tests-started and removed tests-pending labels Nov 27, 2022

cmsbuild added tests-rejected code-checks-pending tests-pending and removed tests-started tests-rejected code-checks-approved labels Nov 27, 2022

cmsbuild added code-checks-approved and removed code-checks-pending labels Nov 28, 2022

cmsbuild added the tests-approved label Dec 3, 2022

cmsbuild added tests-started and removed tests-approved labels Dec 3, 2022

cmsbuild added tests-approved and removed tests-started labels Dec 3, 2022

cmsbuild added xpog-approved and removed xpog-pending labels Dec 4, 2022

swertz mentioned this pull request Dec 4, 2022

[12_6_X] Nano: fix and additions to collection matching, remove cross-cleaning #40234

Merged

cmsbuild added fully-signed reconstruction-approved and removed reconstruction-pending pending-signatures labels Dec 5, 2022

cmsbuild added orp-approved and removed orp-pending labels Dec 6, 2022

cmsbuild merged commit 04a2b8e into cms-sw:master Dec 6, 2022

This was referenced Dec 7, 2022

Updates from Igrof master for Ubuntu which might help with EL8 cms-externals/igprof#5

Merged

[G4VECGEOM] Update clhep to 2.4.6.0, vecgeom to 1.2.1 cms-sw/cmsdist#8211

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Nano: fix and additions to collection matching, remove cross-cleaning #40162

Nano: fix and additions to collection matching, remove cross-cleaning #40162

swertz commented Nov 27, 2022 •

edited

Loading

swertz commented Nov 27, 2022

cmsbuild commented Nov 27, 2022

cmsbuild commented Nov 27, 2022

cmsbuild commented Nov 27, 2022

swertz commented Nov 27, 2022

cmsbuild commented Nov 27, 2022

cmsbuild commented Nov 28, 2022

swertz commented Dec 3, 2022

perrotta commented Dec 3, 2022

cmsbuild commented Dec 3, 2022

swertz commented Dec 4, 2022

mandrenguyen commented Dec 5, 2022

cmsbuild commented Dec 5, 2022

rappoccio commented Dec 6, 2022

smuzaffar commented Dec 6, 2022 •

edited

Loading

rappoccio commented Dec 6, 2022

swertz commented Dec 6, 2022

smuzaffar commented Dec 6, 2022

rappoccio commented Dec 6, 2022

smuzaffar commented Dec 6, 2022

Nano: fix and additions to collection matching, remove cross-cleaning #40162

Nano: fix and additions to collection matching, remove cross-cleaning #40162

Conversation

swertz commented Nov 27, 2022 • edited Loading

PR description:

PR validation:

swertz commented Nov 27, 2022

cmsbuild commented Nov 27, 2022

cmsbuild commented Nov 27, 2022

cmsbuild commented Nov 27, 2022

swertz commented Nov 27, 2022

cmsbuild commented Nov 27, 2022

RelVals-INPUT

RelVals-NANO

Comparison Summary

cmsbuild commented Nov 28, 2022

swertz commented Dec 3, 2022

perrotta commented Dec 3, 2022

cmsbuild commented Dec 3, 2022

Comparison Summary

NANO Comparison Summary

swertz commented Dec 4, 2022

mandrenguyen commented Dec 5, 2022

cmsbuild commented Dec 5, 2022

rappoccio commented Dec 6, 2022

smuzaffar commented Dec 6, 2022 • edited Loading

rappoccio commented Dec 6, 2022

swertz commented Dec 6, 2022

smuzaffar commented Dec 6, 2022

rappoccio commented Dec 6, 2022

smuzaffar commented Dec 6, 2022

swertz commented Nov 27, 2022 •

edited

Loading

smuzaffar commented Dec 6, 2022 •

edited

Loading