ECAL Phase 2 weights method amplitude reconstruction on GPU #37695

ChrisSandever · 2022-04-26T18:21:32Z

PR description:

This PR contains two extra modules in RecoLocalCalo/EcalRecProducers for the ECAL reconstruction on GPU for the Phase 2 upgrade. It modifies 2 other modules with no effect on phase 1 operation.
The new modules are producers, the first of which is a data format converter, which converts phase 2 digis to SoA digis copied onto the GPU device. The other modules is a producer that produces UncalibratedRecHits by reconstructing the amplitudes using the weights method of reconstruction. It is Phase 2 compatible and does the reconstruction on a GPU.
There is an existing module that produces UncalibratedRecHits with the same method for Phase 2 executing the reconstruction on CPU. The UncalibratedRecHits from both modules are virtually identical. Furthermore, the CUDADataFormats/EcalRecHitSoA/interface/EcalUncalibratedRecHit.h data format has been modified to include the AmplitudeError variable to be compatible with the CPU data format.
A switch producer has been added to the phase 2 configuration to be able to run the CPU and GPU algorithms.

PR validation:

The PR passed scram b -tests and runTheMatrix.py -l limited -i all --ibeos.
The configuration was tested with ECAL Phase 2 WF 28234.61.
By adding the gpu modifier to the reconstruction step of WF 28234.61 the GPU branch of the switch producer has been tested.
The UncalibratedRecHits are equivalent for all variables between the CPU and GPU reconstructions.
The timings of the CPU and GPU algorithms have been compared and are similar.

cmsbuild · 2022-04-26T18:29:23Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-37695/29516

This PR adds an extra 32KB to repository

cmsbuild · 2022-04-26T18:29:44Z

A new Pull Request was created by @ChrisSandever (Christopher Rhys Sandever) for master.

It involves the following packages:

CUDADataFormats/EcalRecHitSoA (heterogeneous, reconstruction)
RecoLocalCalo/EcalRecProducers (reconstruction)

@makortel, @slava77, @clacaputo, @cmsbuild, @fwyzard, @jpata can you please review it and eventually sign? Thanks.
@rchatter, @rovere, @argiro, @apsallid, @thomreis, @simonepigazzini this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

jpata · 2022-04-27T14:07:16Z

type ecal

jpata · 2022-04-27T14:07:40Z

enable gpu

jpata · 2022-04-27T14:09:26Z

@cmsbuild please test

jpata · 2022-04-27T14:10:27Z

We have some GPU workflows for Run3 in the matrix, are you planning to perhaps include also one for Phase2, so in the future this can be tested with the bot / in the IB-s?

ChrisSandever · 2022-04-27T14:20:07Z

We also have ECAL Phase 2 development workflows (.61 suffix). We are currently considering to enable the gpu modifier for those. The other option would be to have dedicated WFs like for run3.

cmsbuild · 2022-04-27T18:41:31Z

-1

Failed Tests: RelVals-GPU
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19274f/24264/summary.html
COMMIT: e4e0aea
CMSSW: CMSSW_12_4_X_2022-04-27-1100/slc7_amd64_gcc10
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/37695/24264/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals-GPU

11634.52211634.522_TTbar_14TeV+2021_Patatrack_HCALOnlyGPU+TTbar_14TeV_TuneCP5_GenSim+Digi+RecoNano+HARVESTNano/step2_TTbar_14TeV+2021_Patatrack_HCALOnlyGPU+TTbar_14TeV_TuneCP5_GenSim+Digi+RecoNano+HARVESTNano.log
11634.50611634.506_TTbar_14TeV+2021_Patatrack_PixelOnlyTripletsGPU+TTbar_14TeV_TuneCP5_GenSim+Digi+RecoNano+HARVESTNano/step2_TTbar_14TeV+2021_Patatrack_PixelOnlyTripletsGPU+TTbar_14TeV_TuneCP5_GenSim+Digi+RecoNano+HARVESTNano.log
11634.51211634.512_TTbar_14TeV+2021_Patatrack_ECALOnlyGPU+TTbar_14TeV_TuneCP5_GenSim+Digi+RecoNano+HARVESTNano/step2_TTbar_14TeV+2021_Patatrack_ECALOnlyGPU+TTbar_14TeV_TuneCP5_GenSim+Digi+RecoNano+HARVESTNano.log

Comparison Summary

There are some workflows for which there are errors in the baseline:
1001.2 step 2
The results for the comparisons for these workflows could be incomplete
This means most likely that the IB is having errors in the relvals.The error does NOT come from this pull request

Summary:

No significant changes to the logs found
Reco comparison results: 4 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3695434
DQMHistoTests: Total failures: 8
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3695404
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
Checked 205 log files, 45 edm output root files, 49 DQM output files
TriggerResults: no differences found

fwyzard · 2022-04-28T09:50:48Z

The changes seem to break the HLT running on GPU:

----- Begin Fatal Exception 27-Apr-2022 14:44:24 UTC-----------------------
An exception of category 'StdException' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 1 stream: 0
   [1] Running path 'MC_PFBTagDeepJet_v1'
   [2] Prefetching for module CaloTowersCreator/'hltTowerMakerForAll'
   [3] Prefetching for module EcalRecHitProducer/'hltEcalRecHit@cuda'
   [4] Prefetching for module EcalUncalibRecHitConvertGPU2CPUFormat/'hltEcalUncalibRecHit@cuda'
   [5] Calling method for module EcalCPUUncalibRecHitProducer/'hltEcalUncalibRecHitSoA'
Exception Message:
A std::exception was thrown.

/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_4_X_2022-04-27-1100/src/RecoLocalCalo/EcalRecProducers/plugins/EcalCPUUncalibRecHitProducer.cc, line 84:
cudaCheck(cudaMemcpyAsync(dest.data(), src, dest.size() * sizeof(type), cudaMemcpyDeviceToHost, ctx.stream()));
cudaErrorInvalidValue: invalid argument
----- End Fatal Exception -------------------------------------------------

Can you look into it ?

ChrisSandever · 2022-04-28T10:28:09Z

The reason for the error is the inclusion of the amplitudeError variable in the converters. We have multiple solutions to this at the moment, one is to include the amplitudeError in the phase 1 GPU multifit module where it should be initialized. The other option is to use the boolean introduced in the converters for turning off EE uncalib rechit production to switch between phase 1 and phase 2 behavior to ignore the amplitudeError if in a Phase 1 WF. Both options have been tested and pass these WFs, we are currently timing them.

thomreis · 2022-04-28T11:46:24Z

@fwyzard is the reason that this ECAL changes also break the PixelOnly and HcalOnly WFs because at the HLT all GPU reconstructions are running and the Hcal/PixelOnly in the WF name just corresponds to the reco step?

jpata · 2022-07-07T07:02:06Z

@cmsbuild please test

cmsbuild · 2022-07-07T10:51:29Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19274f/26038/summary.html
COMMIT: 0820f19
CMSSW: CMSSW_12_5_X_2022-07-06-2300/el8_amd64_gcc10
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/37695/26038/install.sh to create a dev area with all the needed externals and cmssw changes.

GPU Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
Reco comparison had 3 failed jobs
DQMHistoTests: Total files compared: 4
DQMHistoTests: Total histograms compared: 19869
DQMHistoTests: Total failures: 9
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 19860
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 3 files compared)
Checked 12 log files, 9 edm output root files, 4 DQM output files
TriggerResults: no differences found

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 2 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 3654771
DQMHistoTests: Total failures: 8
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3654741
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
Checked 208 log files, 45 edm output root files, 50 DQM output files
TriggerResults: no differences found

fwyzard · 2022-07-13T14:51:33Z

+heterogeneous

fwyzard · 2022-07-13T14:51:51Z

@ChrisSandever did you open an issue to keep track of the pending requests ?

ChrisSandever · 2022-07-13T14:55:27Z

Yes, it can be found here: #38619 (comment)

jpata · 2022-08-01T11:20:16Z

@cmsbuild please test

refreshing the tests before signing (I was off)

cmsbuild · 2022-08-01T15:20:38Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19274f/26560/summary.html
COMMIT: 0820f19
CMSSW: CMSSW_12_5_X_2022-08-01-1100/el8_amd64_gcc10
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/37695/26560/install.sh to create a dev area with all the needed externals and cmssw changes.

GPU Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 4
DQMHistoTests: Total histograms compared: 19876
DQMHistoTests: Total failures: 8
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 19868
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 3 files compared)
Checked 12 log files, 9 edm output root files, 4 DQM output files
TriggerResults: no differences found

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 8 differences found in the comparisons
DQMHistoTests: Total files compared: 51
DQMHistoTests: Total histograms compared: 3669004
DQMHistoTests: Total failures: 14
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3668968
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 50 files compared)
Checked 210 log files, 47 edm output root files, 51 DQM output files
TriggerResults: no differences found

jpata · 2022-08-02T07:50:17Z

RecoLocalCalo/EcalRecProducers/python/ecalUncalibRecHitPhase2_cff.py

+        isPhase2 = cms.bool(True),
+        recHitsLabelGPUEB = cms.InputTag('ecalUncalibRecHitSoA', 'EcalUncalibRecHitsEB'),


I'll just note here that

Suggested change

isPhase2 = cms.bool(True),

recHitsLabelGPUEB = cms.InputTag('ecalUncalibRecHitSoA', 'EcalUncalibRecHitsEB'),

isPhase2 = True,

recHitsLabelGPUEB = ('ecalUncalibRecHitSoA', 'EcalUncalibRecHitsEB'),

would be preferred, but this can be done as a quick follow-up too.

jpata · 2022-08-02T07:52:22Z

+reconstruction

new GPU code for Phase2, no effect on Phase1
follow-up requests documented in Follow up PR for PR #37695 ECAL Phase 2 weights method amplitude reconstruction on GPU #38619
presented at the reco meeting: https://indico.cern.ch/event/1176711/#12-ecal-phase-2-weights-method

cmsbuild · 2022-08-02T07:52:44Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

qliphy · 2022-08-04T01:00:56Z

+1

cmsbuild added this to the CMSSW_12_4_X milestone Apr 26, 2022

cmsbuild added code-checks-pending heterogeneous-pending orp-pending pending-signatures reconstruction-pending tests-pending labels Apr 26, 2022

cmsbuild added code-checks-approved and removed code-checks-pending labels Apr 26, 2022

cmsbuild added the ecal label Apr 27, 2022

cmsbuild added tests-started and removed tests-pending labels Apr 27, 2022

cmsbuild added tests-rejected and removed tests-started labels Apr 27, 2022

ChrisSandever force-pushed the ChrisSPhase2EcalRecoGPU branch from e4e0aea to 887c410 Compare April 29, 2022 10:39

cmsbuild added code-checks-pending tests-pending and removed tests-rejected code-checks-approved labels Apr 29, 2022

ChrisSandever mentioned this pull request Jul 6, 2022

Follow up PR for PR #37695 ECAL Phase 2 weights method amplitude reconstruction on GPU #38619

Open

cmsbuild added tests-started and removed tests-pending labels Jul 7, 2022

cmsbuild added tests-approved and removed tests-started labels Jul 7, 2022

cmsbuild added heterogeneous-approved and removed heterogeneous-pending labels Jul 13, 2022

cmsbuild added tests-started and removed tests-approved labels Aug 1, 2022

cmsbuild added tests-approved and removed tests-started labels Aug 1, 2022

jpata reviewed Aug 2, 2022

View reviewed changes

cmsbuild added fully-signed reconstruction-approved and removed reconstruction-pending pending-signatures labels Aug 2, 2022

cmsbuild added orp-approved and removed orp-pending labels Aug 4, 2022

cmsbuild merged commit f47ea73 into cms-sw:master Aug 4, 2022

cmsbuild mentioned this pull request Aug 4, 2022

Update the autoSkim.py for new HLT menu v1.3 #38933

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ECAL Phase 2 weights method amplitude reconstruction on GPU #37695

ECAL Phase 2 weights method amplitude reconstruction on GPU #37695

ChrisSandever commented Apr 26, 2022

cmsbuild commented Apr 26, 2022

cmsbuild commented Apr 26, 2022

jpata commented Apr 27, 2022

jpata commented Apr 27, 2022

jpata commented Apr 27, 2022

jpata commented Apr 27, 2022

ChrisSandever commented Apr 27, 2022

cmsbuild commented Apr 27, 2022

fwyzard commented Apr 28, 2022

ChrisSandever commented Apr 28, 2022 •

edited

Loading

thomreis commented Apr 28, 2022

jpata commented Jul 7, 2022

cmsbuild commented Jul 7, 2022

fwyzard commented Jul 13, 2022

fwyzard commented Jul 13, 2022

ChrisSandever commented Jul 13, 2022

jpata commented Aug 1, 2022

cmsbuild commented Aug 1, 2022

jpata Aug 2, 2022

jpata commented Aug 2, 2022

cmsbuild commented Aug 2, 2022

qliphy commented Aug 4, 2022

		isPhase2 = cms.bool(True),
		recHitsLabelGPUEB = cms.InputTag('ecalUncalibRecHitSoA', 'EcalUncalibRecHitsEB'),

ECAL Phase 2 weights method amplitude reconstruction on GPU #37695

ECAL Phase 2 weights method amplitude reconstruction on GPU #37695

Conversation

ChrisSandever commented Apr 26, 2022

PR description:

PR validation:

cmsbuild commented Apr 26, 2022

cmsbuild commented Apr 26, 2022

jpata commented Apr 27, 2022

jpata commented Apr 27, 2022

jpata commented Apr 27, 2022

jpata commented Apr 27, 2022

ChrisSandever commented Apr 27, 2022

cmsbuild commented Apr 27, 2022

RelVals-GPU

Comparison Summary

fwyzard commented Apr 28, 2022

ChrisSandever commented Apr 28, 2022 • edited Loading

thomreis commented Apr 28, 2022

jpata commented Jul 7, 2022

cmsbuild commented Jul 7, 2022

GPU Comparison Summary

Comparison Summary

fwyzard commented Jul 13, 2022

fwyzard commented Jul 13, 2022

ChrisSandever commented Jul 13, 2022

jpata commented Aug 1, 2022

cmsbuild commented Aug 1, 2022

GPU Comparison Summary

Comparison Summary

jpata Aug 2, 2022

Choose a reason for hiding this comment

jpata commented Aug 2, 2022

cmsbuild commented Aug 2, 2022

qliphy commented Aug 4, 2022

ChrisSandever commented Apr 28, 2022 •

edited

Loading