Backport DeepMET into CMSSW_10_6_X (original: #29764)

[yongbinfeng]

PR description:

Backport DeepMET into CMSSW_10_6_X, in order to get it included in the next ReMiniAOD campaign. The cmsdist in CMSSW_10_6_X also needs to be updated, please test it together with

cms-sw/cmsdist#6054

The massive changes in DataFormats/PatCandidates/interface/MET.h, DataFormats/PatCandidates/src/MET.cc and PhysicsTools/PatAlgos/plugins/PATMETSlimmer.cc are from code-format checks and most of these are unrelated to DeepMET updates.

PR validation:

tested with runTheMatrix.py -l limited -i all --ibeos, failed one test from Flag_trkPOG_logErrorTooManyClusters, which seems irrelevant to our updates. No clue where the problem comes from yet.

if this PR is a backport please specify the original PR and why you need to backport that PR:

backport: #29764

@steggema @intrepid42

[cmsbuild]

A new Pull Request was created by @yongbinfeng (Yongbin Feng) for CMSSW_10_6_X.

It involves the following packages:

DataFormats/PatCandidates
PhysicsTools/PatAlgos
RecoMET/METPUSubtraction

@perrotta, @jpata, @cmsbuild, @santocch, @slava77 can you please review it and eventually sign? Thanks.
@rappoccio, @gouskos, @hatakeyamak, @emilbols, @peruzzim, @seemasharmafnal, @mmarionncern, @ahinzmann, @smoortga, @jdolen, @ferencek, @rovere, @jdamgov, @nhanvtran, @gkasieczka, @schoef, @mariadalfonso, @clelange, @riga, @JyothsnaKomaragiri, @mverzett, @cbernet, @gpetruc, @andrzejnovak this is something you requested to watch as well.
@silviodonato, @dpiparo you are the release manager for this.

cms-bot commands are listed here

• Backported from First version of DeepMET #29764

[perrotta]

backport of #29764

[perrotta]

please test with cms-sw/cmsdist#6054

[cmsbuild]

The tests are being triggered in jenkins.
Tested with other pull request(s) cms-sw/cmsdist#6054

• CMSSW_10_6_X_2020-07-08-2300/slc7_amd64_gcc700: https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/7810/console Started: 2020/07/09 11:31

[cmsbuild]

-1

Tested at: 9a84389

• Build:

I found compilation error when building:

++ . /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/slc7_amd64_gcc700/external/python/2.7.14-pafccj/etc/profile.d/dependencies-setup.sh
+++ test X/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/slc7_amd64_gcc700/external/gcc/7.0.0-pafccj '!=' X
+++ test X '!=' X
+++ . /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/slc7_amd64_gcc700/external/zlib-x86_64/1.2.11-pafccj/etc/profile.d/init.sh
/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/slc7_amd64_gcc700/external/python/2.7.14-pafccj/etc/profile.d/dependencies-setup.sh: line 2: /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/slc7_amd64_gcc700/external/zlib-x86_64/1.2.11-pafccj/etc/profile.d/init.sh: No such file or directory
error: Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.CiedMp (%prep)


RPM build errors:
Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.CiedMp (%prep)



You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ea167c/7810/summary.html

[yongbinfeng]

could the test be restarted now? it seems from cmsdist test (cms-sw/cmsdist#6054) the build is working fine now.

[slava77]

@cmsbuild please test

[cmsbuild]

Pull request #30612 was updated. @perrotta, @jpata, @cmsbuild, @santocch, @slava77 can you please check and sign again.

[yongbinfeng]

Reco comparison results: 76 differences found in the comparisons

this backport should be done with the new feature enabled using run2_miniAOD_devel modifier

done. deepMETProducer enabled only with run2_miniAOD_devel modifier now. please test it together with cms-sw/cmsdist#6054

[slava77]

@cmsbuild please test

[cmsbuild]

The tests are being triggered in jenkins.
Tested with other pull request(s) cms-sw/cmsdist#6054

• CMSSW_10_6_X_2020-07-14-1100/slc7_amd64_gcc700: https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/7934/console Started: 2020/07/14 18:33

[cmsbuild]

+1
Tested at: e4b4bdb
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ea167c/7934/summary.html
CMSSW: CMSSW_10_6_X_2020-07-14-1100
SCRAM_ARCH: slc7_amd64_gcc700

[cmsbuild]

Comparison job queued.

[cmsbuild]

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ea167c/7934/summary.html

Comparison Summary:

• No significant changes to the logs found
• Reco comparison results: 72 differences found in the comparisons
• DQMHistoTests: Total files compared: 34
• DQMHistoTests: Total histograms compared: 3214618
• DQMHistoTests: Total failures: 1
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3214283
• DQMHistoTests: Total skipped: 334
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 33 files compared)
• Checked 140 log files, 14 edm output root files, 34 DQM output files

[slava77]

+1

for #30612 e4b4bdb

• the backport is acceptable: compared to about 250 line changes in First version of DeepMET #29764 this PR also includes clang-format changes and a few lines needed to enable deepMET only with run2_miniAOD_devel
  • despite being disabled by default, the change in pat::MET enums leads to a change in the size of the corrections_ vector data member. IIUC, there should be no restriction to mix the older shorter and the newer longer version as long as users access just what they need. So, it's should be safe to merge, but given the external and the update in the DataFormats/ files, it will need a new full 10_6_X version (not a patch).
• jenkins tests pass and comparisons with the baseline show only somewhat unavoidable changes in the size of pat::MET object corrections_ vector data member.
• I confirmed in a local test that the jobs run with deepMET enabled using run2_miniAOD_devel and that the deepMET modules are indeed not running in the current default

this should be merged with cms-sw/cmsdist#6054

[santocch]

+1

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next CMSSW_10_6_X IBs (tests are also fine) and once validation in the development release cycle CMSSW_11_2_X is complete. This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo (and backports should be raised in the release meeting by the corresponding L2)

[silviodonato]

+1