Ecal DPG - Deploy new software (based on BDT) to recover isolated dead channels… #27175

nancymarinelli · 2019-06-11T16:41:20Z

… in the Ecal barrel
@taroni

PR description:

A new algorithm, based on BDT energy regression is implemented for recovering isolated dead channels in the ECAL barrel

PR validation:

The functionality of the PR has been tested in the 11_0_x IB 06-09-2300. The physics validation has been widely done in 10_2_x and presented in many meetings

cmsbuild · 2019-06-11T16:41:46Z

The code-checks are being triggered in jenkins.

cmsbuild · 2019-06-11T16:47:59Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-27175/10329

This PR adds an extra 28KB to repository

cmsbuild · 2019-06-11T16:48:26Z

A new Pull Request was created by @nancymarinelli for master.

It involves the following packages:

RecoLocalCalo/EcalDeadChannelRecoveryAlgos
RecoLocalCalo/EcalRecProducers

@perrotta, @cmsbuild, @slava77 can you please review it and eventually sign? Thanks.
@argiro this is something you requested to watch as well.
@davidlange6, @slava77, @fabiocos you are the release manager for this.

cms-bot commands are listed here

nancymarinelli · 2019-06-11T17:19:23Z

A new Pull Request was created by @nancymarinelli for master.

It involves the following packages:

RecoLocalCalo/EcalDeadChannelRecoveryAlgos
RecoLocalCalo/EcalRecProducers

@perrotta, @cmsbuild, @slava77 can you please review it and eventually sign? Thanks.
@argiro this is something you requested to watch as well.
@davidlange6, @slava77, @fabiocos you are the release manager for this.

cms-bot commands are listed here

@taroni

slava77 · 2019-06-11T17:23:12Z

RecoLocalCalo/EcalRecProducers/python/ecalRecHit_cfi.py

+    singleChannelRecoveryThreshold = cms.double(0.70),##default 8
+    sum8ChannelRecoveryThreshold = cms.double(0.),##default 8
+    bdtWeightFileNoCracks = cms.FileInPath("RecoLocalCalo/EcalDeadChannelRecoveryAlgos/data/TMVARegression_trainingWithAliveCrystalsAllRH_8GT700MeV_RelEne_noCracks_rightTAG_data_BDTG.weights.xml"),
+    bdtWeightFileCracks = cms.FileInPath("RecoLocalCalo/EcalDeadChannelRecoveryAlgos/data/TMVARegression_trainingWithAliveCrystalsAllRH_8GT700MeV_RelEne_onlyCracks_rightTAG_data_BDTG.weights.xml"),


IIUC, these files are not available yet in a cms-data repo pull request.
Please also check if the name can be shortened.
It looks like noCracks.xml and onlyCracks.xml perhaps with some "v1" for future version tracking is enough.

cmsbuild · 2019-06-12T10:26:42Z

The code-checks are being triggered in jenkins.

cmsbuild · 2019-06-12T10:31:43Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-27175/10335

This PR adds an extra 32KB to repository

cmsbuild · 2019-06-12T10:32:15Z

Pull request #27175 was updated. @perrotta, @cmsbuild, @slava77 can you please check and sign again.

mrodozov · 2019-06-12T13:17:34Z

when your PR is ready to be tested along with it's related cms-data PR
use the following without the quotes
please test with https://github.com/cms-data/RecoLocalCalo-EcalDeadChannelRecoveryAlgos/pull/1, https://github.com/cms-sw/cmsdist/pull/5037
We'll try to remove the cmsdist PR tag gimmick soon (but we're not there yet)
Since the new data PR only adds new files we might merge it and if we do I'll update my comment here

nancymarinelli · 2019-06-12T13:56:26Z

please test with cms-data/RecoLocalCalo-EcalDeadChannelRecoveryAlgos#1, cms-sw/cmsdist#5037

slava77 · 2019-06-12T14:04:21Z

@cmsbuild please test with cms-data/RecoLocalCalo-EcalDeadChannelRecoveryAlgos#1

@mrodozov we were able to test with just cms-data in some recent PRs. Is the cmsdist required again?

cmsbuild · 2019-06-12T14:04:48Z

The tests are being triggered in jenkins.
Tested with other pull request(s) cms-data/RecoLocalCalo-EcalDeadChannelRecoveryAlgos#1
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/915/console Started: 2019/06/12 16:56

cmsbuild · 2019-06-12T14:06:29Z

ERROR: There was an issue generating parameters for
cmsBuild '--source' flag for spec file data-RecoLocalCalo-EcalDeadChannelRecoveryAlgos from cms-data/RecoLocalCalo-EcalDeadChannelRecoveryAlgos repo.

cmsbuild · 2019-07-31T14:58:19Z

Pull request #27175 was updated. @perrotta, @cmsbuild, @slava77 can you please check and sign again.

slava77 · 2019-07-31T15:07:39Z

please test with cms-data/RecoLocalCalo-EcalDeadChannelRecoveryAlgos#1

@nancymarinelli thank you for the quick follow up to the review comments.

cmsbuild · 2019-07-31T15:08:20Z

The tests are being triggered in jenkins.
Tested with other pull request(s) cms-data/RecoLocalCalo-EcalDeadChannelRecoveryAlgos#1
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/1746/console Started: 2019/07/31 17:09

cmsbuild · 2019-07-31T17:24:17Z

+1
Tested at: edab6bc
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-9d7fd8/1746/summary.html

cmsbuild · 2019-07-31T17:24:26Z

Comparison job queued.

cmsbuild · 2019-07-31T19:23:14Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-9d7fd8/1746/summary.html

@slava77 comparisons for the following workflows were not done due to missing matrix map:

/build/cmsbld/jenkins/workspace/compare-root-files-short-matrix/results/JR-comparison/PR-9d7fd8/22034.0_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2026D46_GenSimHLBeamSpotFull14+DigiFullTrigger_2026D46+RecoFullGlobal_2026D46+HARVESTFullGlobal_2026D46

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 32
DQMHistoTests: Total histograms compared: 2484130
DQMHistoTests: Total failures: 1
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2483812
DQMHistoTests: Total skipped: 317
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 31 files compared)
Checked 132 log files, 14 edm output root files, 32 DQM output files

slava77 · 2019-08-01T18:26:49Z

+1

for #27175 edab6bc

code changes are in line with the PR description and the follow up review
jenkins tests pass and comparisons with the baseline show no differences
- the default behavior is no change
- the case of running with the recovery method enabled was tested earlier with 58ca071 and appears to be OK based on the jenkins comparisons

cmsbuild · 2019-08-01T18:27:17Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @davidlange6, @slava77, @smuzaffar, @fabiocos (and backports should be raised in the release meeting by the corresponding L2)

kpedro88 · 2019-08-02T15:40:00Z

+1

cmsbuild added this to the CMSSW_11_0_X milestone Jun 11, 2019

cmsbuild added code-checks-pending comparison-pending orp-pending pending-signatures reconstruction-pending tests-pending labels Jun 11, 2019

cmsbuild added code-checks-approved and removed code-checks-pending labels Jun 11, 2019

slava77 reviewed Jun 11, 2019

View reviewed changes

taroni mentioned this pull request Jun 12, 2019

adding weight files for BDT regression used in deadChannel recovery cms-data/RecoLocalCalo-EcalDeadChannelRecoveryAlgos#1

Merged

cmsbuild added code-checks-pending and removed code-checks-approved labels Jun 12, 2019

cmsbuild added code-checks-approved and removed code-checks-pending labels Jun 12, 2019

cmsbuild added requires-external tests-started and removed tests-pending labels Jun 12, 2019

cmsbuild added code-checks-approved and removed code-checks-pending labels Jul 31, 2019

cmsbuild added requires-external tests-started and removed tests-pending labels Jul 31, 2019

slava77 mentioned this pull request Jul 31, 2019

remove EcalDeadChannelRecoveryNN and related from CMSSW code base #27657

Closed

cmsbuild added tests-approved and removed tests-started labels Jul 31, 2019

cmsbuild added comparison-available and removed comparison-pending labels Jul 31, 2019

cmsbuild added fully-signed reconstruction-approved and removed pending-signatures reconstruction-pending labels Aug 1, 2019

cmsbuild added orp-approved and removed orp-pending labels Aug 2, 2019

cmsbuild merged commit ffabcf0 into cms-sw:master Aug 2, 2019

perrotta mentioned this pull request Sep 13, 2019

Remove EcalDeadChannelRecoveryNN #27986

Merged

guitargeek mentioned this pull request Sep 20, 2019

Migrate EcalDeadChannelRecoveryBDTG to GBRForest #28040

Closed

rchatter mentioned this pull request Jan 30, 2020

Ecal DPG - Backporting new software[PR 27175] to recover isolated new channels #28831

Merged

rchatter mentioned this pull request Feb 25, 2020

Backport: ECAL cluster seed flags and ability to apply updated seeding thresholds from GT #29010

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ecal DPG - Deploy new software (based on BDT) to recover isolated dead channels… #27175

Ecal DPG - Deploy new software (based on BDT) to recover isolated dead channels… #27175

nancymarinelli commented Jun 11, 2019

cmsbuild commented Jun 11, 2019

cmsbuild commented Jun 11, 2019

cmsbuild commented Jun 11, 2019

nancymarinelli commented Jun 11, 2019

slava77 Jun 11, 2019

cmsbuild commented Jun 12, 2019

cmsbuild commented Jun 12, 2019

cmsbuild commented Jun 12, 2019

mrodozov commented Jun 12, 2019

nancymarinelli commented Jun 12, 2019

slava77 commented Jun 12, 2019

cmsbuild commented Jun 12, 2019 •

edited

Loading

cmsbuild commented Jun 12, 2019

cmsbuild commented Jul 31, 2019

slava77 commented Jul 31, 2019

cmsbuild commented Jul 31, 2019 •

edited

Loading

cmsbuild commented Jul 31, 2019

cmsbuild commented Jul 31, 2019

cmsbuild commented Jul 31, 2019

slava77 commented Aug 1, 2019

cmsbuild commented Aug 1, 2019

kpedro88 commented Aug 2, 2019

Ecal DPG - Deploy new software (based on BDT) to recover isolated dead channels… #27175

Ecal DPG - Deploy new software (based on BDT) to recover isolated dead channels… #27175

Conversation

nancymarinelli commented Jun 11, 2019

PR description:

PR validation:

cmsbuild commented Jun 11, 2019

cmsbuild commented Jun 11, 2019

cmsbuild commented Jun 11, 2019

nancymarinelli commented Jun 11, 2019

slava77 Jun 11, 2019

Choose a reason for hiding this comment

cmsbuild commented Jun 12, 2019

cmsbuild commented Jun 12, 2019

cmsbuild commented Jun 12, 2019

mrodozov commented Jun 12, 2019

nancymarinelli commented Jun 12, 2019

slava77 commented Jun 12, 2019

cmsbuild commented Jun 12, 2019 • edited Loading

cmsbuild commented Jun 12, 2019

cmsbuild commented Jul 31, 2019

slava77 commented Jul 31, 2019

cmsbuild commented Jul 31, 2019 • edited Loading

cmsbuild commented Jul 31, 2019

cmsbuild commented Jul 31, 2019

cmsbuild commented Jul 31, 2019

slava77 commented Aug 1, 2019

cmsbuild commented Aug 1, 2019

kpedro88 commented Aug 2, 2019

cmsbuild commented Jun 12, 2019 •

edited

Loading

cmsbuild commented Jul 31, 2019 •

edited

Loading