Adding the Ecal Endcaps to the ML based online ECAL DQM #41175

abhih1 · 2023-03-24T13:03:37Z

PR description:

This PR introduces the Ecal Endcaps into the autoencoder-based online ECAL DQM feature, which was implemented for EB in #35990.
Separate Autoencoder (AE) models with ResNet architecture are trained for EE+ and EE-, apart from the model for EB, on certified good data (digi occupancy) from 2018 runs.

On giving an input occupancy map to the AE, the encoder part of the AE encodes and learns the features and the decoder reconstructs the data from the encoded latent space to match the input as closely as possible. The reconstruction loss is then calculated, which is a mean squared error (MSE) between the input and output images at a tower level. Thus given an anomalous tower, the AE which has learnt the features of the good data will have a hard time reconstructing it and give a higher loss on the anomaly than on the good towers. A quality threshold is then applied on this loss map which marks it as Good or Bad, which is then stored as an ML quality summary plot.
New correction factors are derived from 2022 collisions data to use in the pre-processing and inference, which follows the same steps as used for EB.

This PR thus introduces ML Quality summary plots for EE- and EE+, along with Loss Map and reconstructed occupancy maps from the AE.
It also introduces a trend plot to monitor the no. of bad towers flagged by the AE per lumisection in a run, as well as the map of these bad towers in an occupancy-like plot. This would be very helpful in monitoring per lumisection behaviour of bad towers/channels.

Please note that this PR should be tested along with the files added to cms-data/DQM-EcalMonitorClient#3

PR validation:

The code was validated by running the online Ecal DQM configuration and the resultant plots were examined by uploading the output file to a DQM test gui.
The new plots are confirmed and look reasonable.

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

This is the master PR.
Backport to 13_0_X used in production is here: #41195

cmsbuild · 2023-03-24T13:12:00Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-41175/34851

This PR adds an extra 28KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File DQM/EcalMonitorClient/src/MLClient.cc modified in PR(s): Update to new ONNXRuntime api #41118

cmsbuild · 2023-03-24T13:12:22Z

A new Pull Request was created by @abhih1 (Abhirami Harilal) for master.

It involves the following packages:

DQM/EcalMonitorClient (dqm)
DQM/EcalMonitorTasks (dqm)

@emanueleusai, @cmsbuild, @syuvivida, @rvenditti, @micsucmed, @pmandrik can you please review it and eventually sign? Thanks.
@rchatter, @simonepigazzini, @thomreis, @argiro this is something you requested to watch as well.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.

cms-bot commands are listed here

emanueleusai · 2023-03-27T08:12:26Z

type ecal

emanueleusai · 2023-03-27T08:12:35Z

please test

emanueleusai · 2023-03-27T08:17:23Z

please test with cms-data/DQM-EcalMonitorClient#3

cmsbuild · 2023-03-27T09:18:11Z

-1

Failed Tests: RelVals RelVals-INPUT AddOn
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-6334a6/31609/summary.html
COMMIT: 98b18c4
CMSSW: CMSSW_13_1_X_2023-03-26-2300/el8_amd64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/41175/31609/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals

RelVals-INPUT

AddOn Tests

read312RV

----- Begin Fatal Exception 27-Mar-2023 11:08:38 CEST-----------------------
An exception of category 'ConfigFileReadError' occurred while
   [0] Processing the python configuration file named /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_1_X_2023-03-26-2300/src/Utilities/ReleaseScripts/scripts/read312RV_cfg.py
Exception Message:
 unknown python problem occurred.
ModuleNotFoundError: No module named 'past'

At:
  /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_1_X_2023-03-26-2300/src/FWCore/ParameterSet/python/Types.py(6): <module>
  <frozen importlib._bootstrap>(228): _call_with_frames_removed
  <frozen importlib._bootstrap_external>(850): exec_module
  <frozen importlib._bootstrap>(695): _load_unlocked
  <frozen importlib._bootstrap>(986): _find_and_load_unlocked
  <frozen importlib._bootstrap>(1007): _find_and_load
  /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_1_X_2023-03-26-2300/src/FWCore/ParameterSet/python/Config.py(15): <module>
  <frozen importlib._bootstrap>(228): _call_with_frames_removed
  <frozen importlib._bootstrap_external>(850): exec_module
  <frozen importlib._bootstrap>(695): _load_unlocked
  <frozen importlib._bootstrap>(986): _find_and_load_unlocked
  <frozen importlib._bootstrap>(1007): _find_and_load
  /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_1_X_2023-03-26-2300/src/Utilities/ReleaseScripts/scripts/read312RV_cfg.py(2): <module>

----- End Fatal Exception -------------------------------------------------

fastsim

[fastsim:1] cmsDriver.py TTbar_8TeV_TuneCUETP8M1_cfi  --conditions auto:run1_mc --fast  -n 100 --eventcontent AODSIM,DQM --relval 100000,1000 -s GEN,SIM,RECOBEFMIX,DIGI:pdigi_valid,L1,DIGI2RAW,L1Reco,RECO,VALIDATION  --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --datatier GEN-SIM-DIGI-RECO,DQMIO --beamspot Realistic8TeVCollision : FAILED - elapsed time: 0 sec (ended on Mon Mar 27 11:08:39 2023) - exit: 256

fastsim1

[fastsim1:1] cmsDriver.py TTbar_13TeV_TuneCUETP8M1_cfi --conditions auto:run2_mc_l1stage1 --fast  -n 100 --eventcontent AODSIM,DQM --relval 100000,1000 -s GEN,SIM,RECOBEFMIX,DIGI:pdigi_valid,L1,DIGI2RAW,L1Reco,RECO,VALIDATION  --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --datatier GEN-SIM-DIGI-RECO,DQMIO --beamspot NominalCollision2015 --era Run2_25ns : FAILED - elapsed time: 1 sec (ended on Mon Mar 27 11:08:42 2023) - exit: 256

Expand to see more addon errors ...

abhih1 · 2023-03-27T11:58:20Z

The errors seem to be unrelated to the changes in the PR.

perrotta · 2023-03-27T13:23:47Z

please test with cms-data/DQM-EcalMonitorClient#3

cmsbuild · 2023-03-27T17:24:38Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-6334a6/31614/summary.html
COMMIT: 98b18c4
CMSSW: CMSSW_13_1_X_2023-03-27-1100/el8_amd64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/41175/31614/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-6334a6/31614/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-6334a6/31614/git-merge-result

Comparison Summary

Summary:

You potentially removed 6 lines from the logs
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 143 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3554236
DQMHistoTests: Total failures: 11
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3554203
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 1125.06 KiB( 48 files compared)
DQMHistoSizes: changed ( 20834.76,... ): 20.921 KiB EcalBarrel/EBOccupancyTask
DQMHistoSizes: changed ( 20834.76,... ): 7.799 KiB EcalEndcap/EEOccupancyTask
DQMHistoSizes: changed ( 11834.0,... ): 3.246 KiB Ecal/Trends
DQMHistoSizes: changed ( 11834.0,... ): -0.360 KiB Physics/NanoAODDQM
DQMHistoSizes: changed ( 13234.0,... ): -0.234 KiB Physics/NanoAODDQM
Checked 213 log files, 164 edm output root files, 49 DQM output files
TriggerResults: no differences found

cmsbuild · 2023-04-17T15:57:50Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-41175/35201

This PR adds an extra 16KB to repository

Code check has found code style and quality issues which could be resolved by applying following patch(s)

code-format:
https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-41175/35201/code-format.patch
e.g. curl -k https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-41175/35201/code-format.patch | patch -p1
You can also run scram build code-format to apply code format directly

cmsbuild · 2023-04-17T16:13:02Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-41175/35203

This PR adds an extra 16KB to repository

cmsbuild · 2023-04-17T16:13:28Z

Pull request #41175 was updated. @emanueleusai, @cmsbuild, @syuvivida, @rvenditti, @micsucmed, @pmandrik can you please check and sign again.

perrotta · 2023-04-17T17:13:21Z

please test

cmsbuild · 2023-04-17T19:45:42Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-6334a6/32005/summary.html
COMMIT: f6a03f3
CMSSW: CMSSW_13_1_X_2023-04-17-1100/el8_amd64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/41175/32005/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially removed 22 lines from the logs
Reco comparison results: 10 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3459609
DQMHistoTests: Total failures: 12
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3459575
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 1099.334 KiB( 47 files compared)
DQMHistoSizes: changed ( 1000.0,... ): 20.921 KiB EcalBarrel/EBOccupancyTask
DQMHistoSizes: changed ( 1000.0,... ): 7.799 KiB EcalEndcap/EEOccupancyTask
DQMHistoSizes: changed ( 1000.0,... ): 3.246 KiB Ecal/Trends
Checked 207 log files, 159 edm output root files, 48 DQM output files
TriggerResults: no differences found

emanueleusai · 2023-04-17T23:23:21Z

+1

resign

cmsbuild · 2023-04-17T23:23:48Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

perrotta · 2023-04-18T06:53:57Z

+1

cmsbuild added this to the CMSSW_13_1_X milestone Mar 24, 2023

cmsbuild added dqm-pending pending-signatures tests-pending orp-pending code-checks-pending labels Mar 24, 2023

abhih1 mentioned this pull request Mar 24, 2023

Add Ecal Endcaps onnx models and average occupancy files for ECAL MLDQM cms-data/DQM-EcalMonitorClient#3

Merged

cmsbuild added code-checks-approved and removed code-checks-pending labels Mar 24, 2023

cmsbuild added tests-started ecal and removed tests-pending labels Mar 27, 2023

cmsbuild added the requires-external label Mar 27, 2023

cmsbuild added tests-rejected and removed tests-started labels Mar 27, 2023

aloeliger mentioned this pull request Mar 27, 2023

Added option to trigger on loose HMT showers in EMTF #41185

Merged

abhih1 mentioned this pull request Mar 27, 2023

Adding the Ecal Endcaps to the ML based online ECAL DQM [13_0_X] #41194

Closed

cmsbuild added tests-started and removed tests-rejected labels Mar 27, 2023

This was referenced Mar 27, 2023

Adding the Ecal Endcaps to the ML based online ECAL DQM [13_0_X] #41195

Merged

Add a new MLDQM folder to ECAL workspace in online DQM layout dmwm/deployment#1258

Merged

cmsbuild added the code-checks-pending label Apr 17, 2023

cmsbuild added code-checks-rejected and removed code-checks-pending labels Apr 17, 2023

ML based online DQM adding EE

f6a03f3

abhih1 force-pushed the MLDQMRun3_master branch from 857178e to f6a03f3 Compare April 17, 2023 16:04

cmsbuild added code-checks-pending and removed code-checks-rejected labels Apr 17, 2023

cmsbuild added code-checks-approved and removed code-checks-pending labels Apr 17, 2023

cmsbuild added tests-started and removed tests-pending labels Apr 17, 2023

cmsbuild added tests-approved and removed tests-started labels Apr 17, 2023

cmsbuild added dqm-approved fully-signed and removed dqm-pending pending-signatures labels Apr 17, 2023

cmsbuild added orp-approved and removed orp-pending labels Apr 18, 2023

cmsbuild merged commit 1f28270 into cms-sw:master Apr 18, 2023

cmsbuild mentioned this pull request Apr 18, 2023

Add test for (py3-)torch #41162

Merged

perrotta mentioned this pull request Apr 28, 2023

Update tag for DQM-EcalMonitorClient to V00-03-00 cms-sw/cmsdist#8476

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding the Ecal Endcaps to the ML based online ECAL DQM #41175

Adding the Ecal Endcaps to the ML based online ECAL DQM #41175

abhih1 commented Mar 24, 2023 •

edited

Loading

cmsbuild commented Mar 24, 2023

cmsbuild commented Mar 24, 2023

emanueleusai commented Mar 27, 2023

emanueleusai commented Mar 27, 2023

emanueleusai commented Mar 27, 2023

cmsbuild commented Mar 27, 2023

abhih1 commented Mar 27, 2023

perrotta commented Mar 27, 2023

cmsbuild commented Mar 27, 2023

cmsbuild commented Apr 17, 2023

cmsbuild commented Apr 17, 2023

cmsbuild commented Apr 17, 2023

perrotta commented Apr 17, 2023

cmsbuild commented Apr 17, 2023

emanueleusai commented Apr 17, 2023

cmsbuild commented Apr 17, 2023

perrotta commented Apr 18, 2023

Adding the Ecal Endcaps to the ML based online ECAL DQM #41175

Adding the Ecal Endcaps to the ML based online ECAL DQM #41175

Conversation

abhih1 commented Mar 24, 2023 • edited Loading

PR description:

PR validation:

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

cmsbuild commented Mar 24, 2023

cmsbuild commented Mar 24, 2023

emanueleusai commented Mar 27, 2023

emanueleusai commented Mar 27, 2023

emanueleusai commented Mar 27, 2023

cmsbuild commented Mar 27, 2023

RelVals

RelVals-INPUT

AddOn Tests

abhih1 commented Mar 27, 2023

perrotta commented Mar 27, 2023

cmsbuild commented Mar 27, 2023

Comparison Summary

cmsbuild commented Apr 17, 2023

cmsbuild commented Apr 17, 2023

cmsbuild commented Apr 17, 2023

perrotta commented Apr 17, 2023

cmsbuild commented Apr 17, 2023

Comparison Summary

emanueleusai commented Apr 17, 2023

cmsbuild commented Apr 17, 2023

perrotta commented Apr 18, 2023

abhih1 commented Mar 24, 2023 •

edited

Loading