Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[14_0_X] Using explicit lazy parsing & concrete data types in SimpleFlatTableProducer #44807

Merged
merged 3 commits into from
Apr 24, 2024
Merged

[14_0_X] Using explicit lazy parsing & concrete data types in SimpleFlatTableProducer #44807

merged 3 commits into from
Apr 24, 2024

Conversation

hqucms
Copy link
Contributor

@hqucms hqucms commented Apr 23, 2024

backport of #44782

PR description:

Building on the implementation by @Dr15Jones in #44575 and some discussions there, I modified the usage of SimpleFlatTableProducer to read concrete data types instead of the base reco::Candidate. This avoids the large number of lazyEval = True previously needed in #44575.

PR validation:

Tested for all NANO workflows:

runTheMatrix.py --ibeos --nEvents 100 -w nano -i all -l 2500.0,2500.001,2500.002,2500.01,2500.011,2500.012,2500.1,2500.2,2500.21,2500.211,2500.3,2500.301,2500.31,2500.311,2500.312,2500.313,2500.32,2500.321,2500.322,2500.323,2500.324,2500.325,2500.326,2500.327,2500.4,2500.401,2500.402,2500.403,2500.404,2500.405,2500.5,2500.51

Dr15Jones and others added 3 commits April 23, 2024 00:22
@Dr15Jones @hqucms
Explicit setting of lazy parsing in SimpleFlatTableProducer
16ea729 
The T0 had a failure from the string parser which we have previously
attributed to using the lazy option of the string parser.
Added option to turn on lazy parsing from the configuration. This
is an 'opt' in since lazy parsing can be problematic so best to
explicitly state it must be used.
@hqucms
Use explicit types for SimpleFlatTableProducer.
7e564e8
@hqucms
Split plugins into multiple files.
e4328e7
@cmsbuild cmsbuild added this to the CMSSW_14_0_X milestone Apr 23, 2024
@cmsbuild
Copy link
Contributor

cmsbuild commented Apr 23, 2024 •
edited
Loading

A new Pull Request was created by @hqucms for CMSSW_14_0_X.

It involves the following packages:

  • CalibTracker/SiStripCommon (alca)
  • DPGAnalysis/MuonTools (xpog)
  • DataFormats/Candidate (reconstruction)
  • PhysicsTools/NanoAOD (xpog)

@saumyaphor4252, @cmsbuild, @hqucms, @perrotta, @mandrenguyen, @consuegs, @jfernan2, @vlimant can you please review it and eventually sign? Thanks.
@gbenelli, @tocheng, @missirol, @rovere, @robervalwalsh, @rsreds, @battibass, @yuanchao, @mmusich, @gpetruc, @echabert, @AnnikaStein this is something you requested to watch as well.
@rappoccio, @antoniovilela, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

@cmsbuild
Copy link
Contributor

cmsbuild commented Apr 23, 2024 •
edited
Loading

cms-bot internal usage

@hqucms
Copy link
Contributor Author

hqucms commented Apr 23, 2024

enable nano

@hqucms
Copy link
Contributor Author

hqucms commented Apr 23, 2024

please test

@cmsbuild
Copy link
Contributor

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-3079e2/39016/summary.html
COMMIT: e4328e7
CMSSW: CMSSW_14_0_X_2024-04-22-2300/el8_amd64_gcc12
Additional Tests: NANO
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/44807/39016/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

  • You potentially added 85 lines to the logs
  • Reco comparison results: 56 differences found in the comparisons
  • DQMHistoTests: Total files compared: 48
  • DQMHistoTests: Total histograms compared: 3319088
  • DQMHistoTests: Total failures: 9
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 3319059
  • DQMHistoTests: Total skipped: 20
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
  • Checked 202 log files, 165 edm output root files, 48 DQM output files
  • TriggerResults: no differences found

NANO Comparison Summary

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 0 differences found in the comparisons
  • DQMHistoTests: Total files compared: 15
  • DQMHistoTests: Total histograms compared: 17023
  • DQMHistoTests: Total failures: 0
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 17023
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 14 files compared)
  • Checked 49 log files, 29 edm output root files, 15 DQM output files

Nano size comparison Summary:

Sample kb/ev ref kb/ev diff kb/ev ev/s/thd ref ev/s/thd diff rate mem/thd ref mem/thd
2500.0 2.783 2.783 0.000 ( +0.0% ) 4.00 3.92 +2.0% 2.161 2.253
2500.001 2.897 2.897 0.000 ( +0.0% ) 3.55 3.47 +2.5% 2.562 2.672
2500.002 2.843 2.843 0.000 ( +0.0% ) 3.69 3.64 +1.3% 2.550 2.670
2500.01 1.446 1.446 0.000 ( +0.0% ) 7.01 6.92 +1.4% 2.271 2.373
2500.011 1.906 1.906 0.000 ( +0.0% ) 3.72 3.69 +0.8% 2.431 2.562
2500.012 1.761 1.761 0.000 ( +0.0% ) 5.34 5.39 -0.9% 2.336 2.466
2500.1 2.354 2.354 0.000 ( +0.0% ) 5.22 5.08 +2.8% 2.079 2.077
2500.2 2.459 2.459 0.000 ( +0.0% ) 5.99 5.77 +3.8% 1.989 1.995
2500.21 1.286 1.286 0.000 ( +0.0% ) 4.26 4.20 +1.5% 2.299 2.296
2500.211 1.668 1.668 0.000 ( +0.0% ) 3.94 3.71 +6.1% 2.370 2.375
2500.3 2.229 2.229 0.000 ( +0.0% ) 12.00 11.42 +5.1% 1.975 1.979
2500.301 2.833 2.833 0.000 ( +0.0% ) 10.99 9.54 +15.2% 1.963 1.966
2500.31 1.348 1.348 0.000 ( +0.0% ) 20.12 18.59 +8.2% 2.357 2.383
2500.311 1.757 1.757 0.000 ( +0.0% ) 15.85 12.93 +22.6% 2.447 2.455
2500.312 7.164 7.164 0.000 ( +0.0% ) 1.58 1.42 +11.8% 1.692 1.697
2500.313 1.568 1.568 0.000 ( +0.0% ) 9.25 6.70 +38.1% 1.052 1.039
2500.314 1.236 1.236 0.000 ( +0.0% ) 14.64 13.26 +10.4% 2.225 2.292
2500.315 1.874 1.874 0.000 ( +0.0% ) 15.67 12.90 +21.4% 2.345 2.347
2500.316 3.342 3.342 0.000 ( +0.0% ) 2.12 2.01 +5.5% 2.307 2.306
2500.317 1.811 1.811 0.000 ( +0.0% ) 15.07 13.71 +9.9% 2.192 2.365
2500.318 4.136 4.136 0.000 ( +0.0% ) 5.85 5.46 +7.2% 2.181 2.351
2500.4 2.374 2.374 0.000 ( +0.0% ) 11.89 11.28 +5.4% 1.859 1.943
2500.401 1.885 1.885 0.000 ( +0.0% ) 10.81 9.34 +15.8% 1.747 1.865
2500.402 2.950 2.950 0.000 ( +0.0% ) 10.59 9.19 +15.3% 1.723 1.879
2500.403 5.474 5.474 0.000 ( +0.0% ) 1.49 1.41 +6.1% 1.954 1.968
2500.404 2.860 2.860 0.000 ( +0.0% ) 10.54 9.37 +12.5% 1.835 1.901
2500.405 8.700 8.700 0.000 ( +0.0% ) 3.66 3.31 +10.7% 1.763 1.786
2500.5 5.194 5.194 0.000 ( +0.0% ) 16.44 15.79 +4.1% 1.553 1.704
2500.51 9.120 9.120 0.000 ( +0.0% ) 9.97 9.58 +4.1% 1.516 1.649

@hqucms
Copy link
Contributor Author

hqucms commented Apr 23, 2024

+1

@jfernan2
Copy link
Contributor

+1

@perrotta
Copy link
Contributor

+alca

  • Updates in CalibTracker/SiStripCommon/plugins/SealModules.cc look correct
  • This being technical, even if the master is not yet merged, I can sign for alca/db: then the release managers will decide when to merge

@cmsbuild
Copy link
Contributor

This pull request is fully signed and it will be integrated in one of the next CMSSW_14_0_X IBs (tests are also fine) and once validation in the development release cycle CMSSW_14_1_X is complete. This pull request will now be reviewed by the release team before it's merged. @sextonkennedy, @rappoccio, @antoniovilela (and backports should be raised in the release meeting by the corresponding L2)

@rappoccio
Copy link
Contributor

+1

@cmsbuild cmsbuild merged commit f6ecc6e into cms-sw:CMSSW_14_0_X Apr 24, 2024
12 checks passed
@antoniovilela
Copy link
Contributor

@cms-sw/xpog-l2 @hqucms

We need a quick fix for 14_1_X and 14_0_X for the ProductNotFound error in RelVals:

https://cmssdt.cern.ch/SDT/cgi-bin/logreader/el8_amd64_gcc12/CMSSW_14_0_X_2024-04-24-2300/pyRelValMatrixLogs/run/546.0_DYToLL_M-50_13TeV_pythia8/step2_DYToLL_M-50_13TeV_pythia8.log#/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
alca-approved backport-ok fully-signed orp-approved reconstruction-approved tests-approved trk xpog-approved
Projects
None yet
Milestone
CMSSW_14_0_X
Development

Successfully merging this pull request may close these issues.
7 participants
@hqucms @cmsbuild @jfernan2 @perrotta @rappoccio @antoniovilela @Dr15Jones