Improve error handling in GPU callbacks [14.0.x]

[fwyzard]

PR description:

Update the callbacks used on the CUDA and HIP/ROCm backends to match the original CUDA implementation: in case of asynchronous errors, throw-catch an exception to let GDB intercept it, and propagate the exception to the framework.

PR validation:

Unit tests ran.

Backport status

Backport of #44476 to 14.0.x for data taking.

[fwyzard]

backport #44476

[fwyzard]

enable gpu

[fwyzard]

please test

[cmsbuild]

A new Pull Request was created by @fwyzard for CMSSW_14_0_X.

It involves the following packages:

• HeterogeneousCore/AlpakaCore (heterogeneous)
• HeterogeneousCore/AlpakaInterface (heterogeneous)

@fwyzard, @makortel can you please review it and eventually sign? Thanks.
@makortel, @rovere, @missirol this is something you requested to watch as well.
@antoniovilela, @rappoccio, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

• Backported from Improve error handling in GPU callbacks #44476

[cmsbuild]

cms-bot internal usage

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0a5758/38271/summary.html
COMMIT: 8c3b153
CMSSW: CMSSW_14_0_X_2024-03-19-2300/el8_amd64_gcc12
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/44477/38271/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 38 differences found in the comparisons
• DQMHistoTests: Total files compared: 49
• DQMHistoTests: Total histograms compared: 3346212
• DQMHistoTests: Total failures: 3
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3346187
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
• Checked 205 log files, 166 edm output root files, 49 DQM output files
• TriggerResults: no differences found

GPU Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 11 differences found in the comparisons
• DQMHistoTests: Total files compared: 3
• DQMHistoTests: Total histograms compared: 39740
• DQMHistoTests: Total failures: 500
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 39240
• DQMHistoTests: Total skipped: 0
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 2 files compared)
• Checked 8 log files, 10 edm output root files, 3 DQM output files
• TriggerResults: no differences found

[cmsbuild]

Pull request #44477 was updated. @cmsbuild, @fwyzard, @makortel can you please check and sign again.

[fwyzard]

please test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0a5758/38299/summary.html
COMMIT: 127bf32
CMSSW: CMSSW_14_0_X_2024-03-20-1100/el8_amd64_gcc12
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/44477/38299/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially added 95 lines to the logs
• Reco comparison results: 62 differences found in the comparisons
• DQMHistoTests: Total files compared: 49
• DQMHistoTests: Total histograms compared: 3346212
• DQMHistoTests: Total failures: 1358
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3344832
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
• Checked 205 log files, 166 edm output root files, 49 DQM output files
• TriggerResults: no differences found

GPU Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 20 differences found in the comparisons
• DQMHistoTests: Total files compared: 3
• DQMHistoTests: Total histograms compared: 39740
• DQMHistoTests: Total failures: 662
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 39078
• DQMHistoTests: Total skipped: 0
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 2 files compared)
• Checked 8 log files, 10 edm output root files, 3 DQM output files
• TriggerResults: no differences found

[fwyzard]

+heterogeneous

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next CMSSW_14_0_X IBs (tests are also fine) and once validation in the development release cycle CMSSW_14_1_X is complete. This pull request will now be reviewed by the release team before it's merged. @rappoccio, @sextonkennedy, @antoniovilela (and backports should be raised in the release meeting by the corresponding L2)

[antoniovilela]

+1