DQM: new DQMGlobalEDAnalyzer #28407

schneiml · 2019-11-15T16:07:54Z

PR description:

This PR imports more features from the "new DQMStore" project:

A new DQMEDAnalyzer base class, now based on EDProducer, like all the other DQM module base classes. This is to make sure that everything that produces DQM histograms actually is known to the EDM framework as a Producer, even if the data still flows through the DQMStore; otherwise, we have no guarantees that modules will run in the correct order.
Some cleanup removing unused declarations in DQMStore.h, including fastmatch. The previous cleanup PRs left these unused.

Update: some things went into #28612, removed here now.

PR validation:

The challenge with this PR is that DQMGlobalEDAnalyzers are primarily used in HLT, and I am not sure how to test them there. For now, HLT configs should reliably crash due to the incorrect module type used in the auto-generated configs; I am not sure how to change this part of the code. Also FastTimerService and ThroughputService need to be checked. Let's see if any of that is picked up in the PR tests, else I'll have to defer that to the HLT experts.

cmsbuild · 2019-11-15T16:08:30Z

The code-checks are being triggered in jenkins.

cmsbuild · 2019-11-15T16:16:54Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-28407/12777

This PR adds an extra 728KB to repository

cmsbuild · 2019-11-15T16:17:29Z

A new Pull Request was created by @schneiml (Marcel Schneider) for master.

It involves the following packages:

Alignment/OfflineValidation
CalibMuon/DTCalibration
CalibTracker/SiStripChannelGain
Calibration/EcalCalibAlgos
Calibration/TkAlCaRecoProducers
DQM/BeamMonitor
DQM/CSCMonitorModule
DQM/DTMonitorClient
DQM/DTMonitorModule
DQM/EcalCommon
DQM/EcalMonitorDbModule
DQM/EcalMonitorTasks
DQM/EcalPreshowerMonitorClient
DQM/EcalPreshowerMonitorModule
DQM/HLTEvF
DQM/HLXMonitor
DQM/HcalTasks
DQM/Integration
DQM/L1TMonitor
DQM/L1TMonitorClient
DQM/Physics
DQM/RPCMonitorClient
DQM/SiPixelCommon
DQM/SiPixelMonitorClient
DQM/SiPixelPhase1Config
DQM/SiPixelPhase1Summary
DQM/SiStripCommissioningClients
DQM/SiStripCommissioningSources
DQM/SiStripMonitorClient
DQM/SiStripMonitorSummary
DQM/TrackingMonitor
DQM/TrackingMonitorClient
DQMOffline/CalibMuon
DQMOffline/EGamma
DQMOffline/Ecal
DQMOffline/JetMET
DQMOffline/L1Trigger
DQMOffline/Muon
DQMOffline/PFTau
DQMOffline/Trigger
DQMServices/ClientConfig
DQMServices/Components
DQMServices/Core
DQMServices/Demo
DQMServices/Examples
GeneratorInterface/RivetInterface
HLTrigger/Timer
HLTriggerOffline/B2G
HLTriggerOffline/Common
HLTriggerOffline/Exotica
HLTriggerOffline/Higgs
HLTriggerOffline/JetMET
HLTriggerOffline/Muon
HLTriggerOffline/SUSYBSM
HLTriggerOffline/Tau
PhysicsTools/NanoAOD
Validation/DTRecHits
Validation/HGCalValidation
Validation/L1T
Validation/RecoB
Validation/RecoMuon
Validation/RecoTau
Validation/RecoTrack

The following packages do not have a category, yet:

DQMServices/Demo
Please create a PR for https://github.com/cms-sw/cms-bot/blob/master/categories_map.py to assign category

@SiewYan, @andrius-k, @schneiml, @Martin-Grunewald, @rekovic, @fioriNTU, @tlampen, @alberto-sanchez, @pohsun, @santocch, @peruzzim, @cmsbuild, @agrohsje, @fwyzard, @efeyazgan, @tocheng, @jfernan2, @qliphy, @benkrikler, @mkirsano, @kmaeshima, @christopheralanwest, @franzoni, @fgolf can you please review it and eventually sign? Thanks.
@wmtford, @erikbutz, @rappoccio, @apsallid, @echabert, @felicepantaleo, @schoef, @robervalwalsh, @emilbols, @kpedro88, @argiro, @HeinerTholen, @alesaggio, @bsunanda, @pfs, @tlampen, @alberto-sanchez, @ahinzmann, @missirol, @threus, @mmusich, @seemasharmafnal, @venturia, @acimmino, @mmarionncern, @kreczko, @sethzenz, @calderona, @makortel, @smoortga, @acaudron, @jhgoh, @lgray, @jdolen, @HuguesBrun, @agrohsje, @cericeci, @ferencek, @pieterdavid, @rociovilar, @vandreev11, @abbiendi, @barvic, @DryRun, @hdelanno, @GiacomoSguazzoni, @rovere, @VinInn, @cseez, @bellan, @nhanvtran, @gkasieczka, @tocheng, @deguio, @jandrea, @mschrode, @idebruyn, @ebrondol, @ptcox, @mtosi, @dgulhan, @clelange, @mkirsano, @jdamgov, @batinkov, @adewit, @trocino, @battibass, @gbenelli, @Fedespring, @JyothsnaKomaragiri, @mverzett, @Martin-Grunewald, @gpetruc, @mariadalfonso, @pvmulder, @andrzejnovak, @folguera, @fioriNTU this is something you requested to watch as well.
@davidlange6, @slava77, @fabiocos you are the release manager for this.

cms-bot commands are listed here

schneiml · 2019-11-15T17:13:54Z

please test

Mostly to see if there is a workflow that blows up due to incorrect configuration, and if the DQMGlobalEDAnalyzer MEs still end up in the output.

cmsbuild · 2019-11-15T17:14:31Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/3497/console Started: 2019/11/15 18:15

cmsbuild · 2019-11-15T19:46:35Z

+1
Tested at: fcd2d6c
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-a20bd6/3497/summary.html

cmsbuild · 2019-11-15T19:46:39Z

Comparison job queued.

cmsbuild · 2019-11-15T21:20:59Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-a20bd6/3497/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 28 differences found in the comparisons
DQMHistoTests: Total files compared: 34
DQMHistoTests: Total histograms compared: 2936360
DQMHistoTests: Total failures: 11
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2936008
DQMHistoTests: Total skipped: 341
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 1464.852 KiB( 33 files compared)
DQMHistoSizes: changed ( 1000.0,... ): 25.259 KiB Egamma/gedPhotonAnalyzer
DQMHistoSizes: changed ( 1000.0,... ): 25.259 KiB Egamma/stdPhotonAnalyzer
DQMHistoSizes: changed ( 1000.0,... ): 5.613 KiB Egamma/Electrons
DQMHistoSizes: changed ( 1000.0,... ): 0.533 KiB Tracking/TrackParameters
DQMHistoSizes: changed ( 1000.0,... ): -0.093 KiB L1T/L1TOccupancy
DQMHistoSizes: changed ( 10024.0,... ): 2.364 KiB BDHadronTracks/JetContent
DQMHistoSizes: changed ( 10024.0,... ): 0.842 KiB ParticleFlow/PFJetValidation
DQMHistoSizes: changed ( 10024.0,... ): 0.451 KiB ParticleFlow/PFMETValidation
DQMHistoSizes: changed ( 10024.0,... ): -0.047 KiB MessageLogger/Errors
DQMHistoSizes: changed ( 10024.0,... ): -0.047 KiB MessageLogger/Warnings
DQMHistoSizes: changed ( 1306.0 ): ...
Checked 147 log files, 16 edm output root files, 34 DQM output files

cmsbuild · 2020-01-07T13:03:54Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-24766c/4155/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 28 differences found in the comparisons
DQMHistoTests: Total files compared: 34
DQMHistoTests: Total histograms compared: 2815513
DQMHistoTests: Total failures: 11
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2815156
DQMHistoTests: Total skipped: 346
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -1.462 KiB( 33 files compared)
DQMHistoSizes: changed ( 10024.0,... ): -0.047 KiB MessageLogger/Errors
DQMHistoSizes: changed ( 10024.0,... ): -0.047 KiB MessageLogger/Warnings
DQMHistoSizes: changed ( 1306.0,... ): -0.035 KiB MessageLogger/Errors
DQMHistoSizes: changed ( 1306.0,... ): -0.035 KiB MessageLogger/Warnings
DQMHistoSizes: changed ( 20034.0,... ): -0.051 KiB MessageLogger/Errors
DQMHistoSizes: changed ( 20034.0,... ): -0.051 KiB MessageLogger/Warnings
Checked 147 log files, 16 edm output root files, 34 DQM output files

silviodonato · 2020-01-07T16:46:03Z

@schneiml shall we wait for #28588 before merging this PR?

schneiml · 2020-01-08T09:06:37Z

@silviodonato no, these two should be completely independent, either can go first.

silviodonato · 2020-01-08T16:03:54Z

please test

cmsbuild · 2020-01-08T16:04:18Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/4168/console Started: 2020/01/08 17:06

cmsbuild · 2020-01-08T17:58:08Z

+1
Tested at: 1dd3a3d
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-a20bd6/4168/summary.html
CMSSW: CMSSW_11_1_X_2020-01-08-1100
SCRAM_ARCH: slc7_amd64_gcc820

cmsbuild · 2020-01-08T17:58:11Z

Comparison job queued.

cmsbuild · 2020-01-08T19:24:36Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-a20bd6/4168/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 28 differences found in the comparisons
DQMHistoTests: Total files compared: 34
DQMHistoTests: Total histograms compared: 2815513
DQMHistoTests: Total failures: 11
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2815156
DQMHistoTests: Total skipped: 346
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -1.462 KiB( 33 files compared)
DQMHistoSizes: changed ( 10024.0,... ): -0.047 KiB MessageLogger/Errors
DQMHistoSizes: changed ( 10024.0,... ): -0.047 KiB MessageLogger/Warnings
DQMHistoSizes: changed ( 1306.0,... ): -0.035 KiB MessageLogger/Errors
DQMHistoSizes: changed ( 1306.0,... ): -0.035 KiB MessageLogger/Warnings
DQMHistoSizes: changed ( 20034.0,... ): -0.051 KiB MessageLogger/Errors
DQMHistoSizes: changed ( 20034.0,... ): -0.051 KiB MessageLogger/Warnings
Checked 147 log files, 16 edm output root files, 34 DQM output files

silviodonato · 2020-01-08T20:22:57Z

+1

cmsbuild added this to the CMSSW_11_0_X milestone Nov 15, 2019

cmsbuild added alca-pending analysis-pending code-checks-pending comparison-pending dqm-pending generators-pending hlt-pending l1-pending new-package-pending orp-pending pending-signatures tests-pending xpog-pending labels Nov 15, 2019

cmsbuild added code-checks-approved and removed code-checks-pending labels Nov 15, 2019

cmsbuild added tests-started and removed tests-pending labels Nov 15, 2019

cmsbuild added tests-approved and removed tests-started labels Nov 15, 2019

cmsbuild added comparison-available and removed comparison-pending labels Nov 15, 2019

cmsbuild added comparison-pending tests-approved and removed comparison-available tests-started labels Jan 7, 2020

cmsbuild added comparison-available and removed comparison-pending labels Jan 7, 2020

cmsbuild added comparison-pending tests-pending and removed comparison-available tests-approved labels Jan 8, 2020

cmsbuild added tests-started and removed requires-external tests-pending labels Jan 8, 2020

cmsbuild added tests-approved and removed tests-started labels Jan 8, 2020

cmsbuild added comparison-available and removed comparison-pending labels Jan 8, 2020

cmsbuild added orp-approved and removed orp-pending labels Jan 8, 2020

cmsbuild merged commit a8d5c49 into cms-sw:master Jan 8, 2020

cmsbuild mentioned this pull request Jan 9, 2020

TestProcessor supports Run and Lumi product testing #28721

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DQM: new DQMGlobalEDAnalyzer #28407

DQM: new DQMGlobalEDAnalyzer #28407

schneiml commented Nov 15, 2019 •

edited

Loading

cmsbuild commented Nov 15, 2019

cmsbuild commented Nov 15, 2019

cmsbuild commented Nov 15, 2019

schneiml commented Nov 15, 2019

cmsbuild commented Nov 15, 2019 •

edited

Loading

cmsbuild commented Nov 15, 2019

cmsbuild commented Nov 15, 2019

cmsbuild commented Nov 15, 2019

cmsbuild commented Jan 7, 2020

silviodonato commented Jan 7, 2020

schneiml commented Jan 8, 2020

silviodonato commented Jan 8, 2020

cmsbuild commented Jan 8, 2020 •

edited

Loading

cmsbuild commented Jan 8, 2020

cmsbuild commented Jan 8, 2020

cmsbuild commented Jan 8, 2020

silviodonato commented Jan 8, 2020

DQM: new DQMGlobalEDAnalyzer #28407

DQM: new DQMGlobalEDAnalyzer #28407

Conversation

schneiml commented Nov 15, 2019 • edited Loading

PR description:

PR validation:

cmsbuild commented Nov 15, 2019

cmsbuild commented Nov 15, 2019

cmsbuild commented Nov 15, 2019

schneiml commented Nov 15, 2019

cmsbuild commented Nov 15, 2019 • edited Loading

cmsbuild commented Nov 15, 2019

cmsbuild commented Nov 15, 2019

cmsbuild commented Nov 15, 2019

cmsbuild commented Jan 7, 2020

silviodonato commented Jan 7, 2020

schneiml commented Jan 8, 2020

silviodonato commented Jan 8, 2020

cmsbuild commented Jan 8, 2020 • edited Loading

cmsbuild commented Jan 8, 2020

cmsbuild commented Jan 8, 2020

cmsbuild commented Jan 8, 2020

silviodonato commented Jan 8, 2020

schneiml commented Nov 15, 2019 •

edited

Loading

cmsbuild commented Nov 15, 2019 •

edited

Loading

cmsbuild commented Jan 8, 2020 •

edited

Loading