Introduce matrix workflows for Alpaka-based Particle Flow

[jsamudio]

PR description:

This PR adds new workflows to test Alpaka-based PF modules in CMSSW. Introduced are:

.431 : HCAL Only Reconstruction using Alpaka-based PF, slimmed for benchmarking
.432 : HCAL Only Reconstruction on GPU using Alpaka-based PF, with DQM and Validation
.433 : HCAL Only Reconstruction on CPU using Alpaka-based PF, with DQM, Validation, and Alpaka vs Legacy PF Cluster comparison
.434 : HCAL Only Reconstruction on GPU using Alpaka-based PF, with DQM, Validation, and Alpaka vs Legacy PF Cluster comparison
.435 : HCAL Only Reconstruction on GPU using Alpaka-based PF (forced CUDA), with DQM, Validation, and Alpaka vs Legacy PF Cluster comparison

• 0.422: Alpaka, HCAL only, portable
• 0.423: Alpaka, HCAL only, portable vs CPU validation
• 0.424: Alpaka, HCAL only, portable profiling
• 0.492 Alpaka, Pixel quadruplets, Full-Reco

PR validation:

All new workflows tested and passing on 12434.

PF cluster Alpaka vs Legacy validation plots from 12434.423 are available at https://hep.baylor.edu/jsamudio/workflowValidation/

Usual runTheMatrix.py -l limited -i all --ibeos check is done and passes, except for step3 failure on 1000.0 from Fatal Root Error: @SUB=TStorageFactorySystem::Unlink which I believe is unrelated to the changes in this PR.

@hatakeyamak @fwyzard

[cmsbuild]

cms-bot internal usage

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-43701/38388

• This PR adds an extra 40KB to repository

• There are other open Pull requests which might conflict with changes you have proposed:

  • File Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py modified in PR(s): Add run3 tracking low pu era #33532, Porting Pixel Tracks to Alpaka [Not to Merge]  #41117, Guard for No Digis for Phase2 and Patatrack NuGun #41255, Pixel Alpaka Migration: Configs and Fixes [VII] #43294, Add Phase-2 workflow with Phase-2 ECal TP, also with ECal component  #43635, Phase2-HGX350A Resolve the scenario for V18 version of HGCal Geometry #43661

[cmsbuild]

A new Pull Request was created by @jsamudio for master.

It involves the following packages:

• Configuration/ProcessModifiers (operations)
• Configuration/PyReleaseValidation (upgrade, pdmv)
• RecoParticleFlow/PFClusterProducer (reconstruction)
• Validation/RecoParticleFlow (dqm)

@AdrianoDee, @syuvivida, @antoniovilela, @mandrenguyen, @subirsarkar, @jfernan2, @miquork, @rvenditti, @sunilUIET, @tjavaid, @cmsbuild, @srimanob, @antoniovagnerini, @nothingface0, @fabiocos, @davidlange6, @rappoccio can you please review it and eventually sign? Thanks.
@felicepantaleo, @seemasharmafnal, @hatakeyamak, @ebrondol, @slomeo, @lgray, @mmarionncern, @makortel, @rovere, @missirol, @Martin-Grunewald, @sameasy, @fabiocos this is something you requested to watch as well.
@sextonkennedy, @rappoccio, @antoniovilela you are the release manager for this.

cms-bot commands are listed here

[srimanob]

Hi @jsamudio
Please update the list of workflows in the README file, https://github.com/cms-sw/cmssw/blob/master/Configuration/PyReleaseValidation/README.md

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-43701/38389

• This PR adds an extra 44KB to repository

• There are other open Pull requests which might conflict with changes you have proposed:

  • File Configuration/PyReleaseValidation/README.md modified in PR(s): Add run3 tracking low pu era #33532, Add Phase-2 workflow with Phase-2 ECal TP, also with ECal component  #43635
  • File Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py modified in PR(s): Add run3 tracking low pu era #33532, Porting Pixel Tracks to Alpaka [Not to Merge]  #41117, Guard for No Digis for Phase2 and Patatrack NuGun #41255, Pixel Alpaka Migration: Configs and Fixes [VII] #43294, Add Phase-2 workflow with Phase-2 ECal TP, also with ECal component  #43635, Phase2-HGX350A Resolve the scenario for V18 version of HGCal Geometry #43661

[cmsbuild]

Pull request #43701 was updated. @cmsbuild, @davidlange6, @nothingface0, @rvenditti, @syuvivida, @AdrianoDee, @antoniovilela, @srimanob, @jfernan2, @mandrenguyen, @subirsarkar, @tjavaid, @fabiocos, @miquork, @rappoccio, @sunilUIET, @antoniovagnerini can you please check and sign again.

[hatakeyamak]

Can somebody trigger jenkins?

[AdrianoDee]

enable gpu

[AdrianoDee]

test parameters:

• workflow_opts_gpu= -w upgrade
• workflows_gpu=12434.431, 12434.432,12434.434,12434.435,12434.594

[AdrianoDee]

please test
(hoping to have nailed the bot parameters at the first attempt)

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-10733b/36811/summary.html
COMMIT: bd881f2
CMSSW: CMSSW_14_0_X_2024-01-12-1100/el8_amd64_gcc12
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/43701/36811/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially added 2 lines to the logs
• Reco comparison results: 8 differences found in the comparisons
• DQMHistoTests: Total files compared: 48
• DQMHistoTests: Total histograms compared: 3247277
• DQMHistoTests: Total failures: 6
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3247249
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
• Checked 200 log files, 161 edm output root files, 48 DQM output files
• TriggerResults: no differences found

GPU Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 11 differences found in the comparisons
• DQMHistoTests: Total files compared: 3
• DQMHistoTests: Total histograms compared: 39740
• DQMHistoTests: Total failures: 508
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 39232
• DQMHistoTests: Total skipped: 0
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 2 files compared)
• Checked 8 log files, 10 edm output root files, 3 DQM output files
• TriggerResults: no differences found

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-10733b/37434/summary.html
COMMIT: 59d98c4
CMSSW: CMSSW_14_1_X_2024-02-13-1100/el8_amd64_gcc12
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/43701/37434/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially added 111 lines to the logs
• Reco comparison results: 67 differences found in the comparisons
• DQMHistoTests: Total files compared: 48
• DQMHistoTests: Total histograms compared: 3248554
• DQMHistoTests: Total failures: 1192
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3247340
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
• Checked 200 log files, 161 edm output root files, 48 DQM output files
• TriggerResults: no differences found

GPU Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 48 differences found in the comparisons
• DQMHistoTests: Total files compared: 3
• DQMHistoTests: Total histograms compared: 39740
• DQMHistoTests: Total failures: 1875
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 37865
• DQMHistoTests: Total skipped: 0
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 2 files compared)
• Checked 8 log files, 10 edm output root files, 3 DQM output files
• TriggerResults: no differences found

[jfernan2]

+1

[AdrianoDee]

+pdmv

[srimanob]

+Upgrade

[fwyzard]

@cms-sw/dqm-l2 sorry for bothering you explicitly, but I would like to get this PR and its backport merged in time for 14.0.0 tomorrow.

Could you confirm that you are aware of it, and are either fine with it, or testing it, or plan to do it soon ?

[antoniovilela]

+1

• Once DQM signs, it will get merged.

[tjavaid]

+1

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will be automatically merged.