Enable "alternative module loading" approach for Alpaka modules #40564

makortel · 2023-01-19T01:26:41Z

PR description:

This PR enables the "alternative module loading" functionality for Alpaka modules (following what I presented in https://indico.cern.ch/event/1208060/#16-portable-configuration-with). In order to use this mechanism, the configuration needs to load HeterogeneousCore.AlpakaCore.ProcessAcceleratorAlpaka_cfi, and define the modules in the configuration with @alpaka postfix in the type name along

# selects at run time whether alpaka_serial_sync::FooProducer or alpaka_cuda_async::FooProducer is used
process.foo = cms.EDProducer("FooProducer@alpaka", ...)

The syntax works for EDProducers, EDAnalyzers, EDFilters, and ESProducers.

The backend can also be set explicitly via

process.foo = cms.EDProducer("FooProducer@alpaka",
    ...,
    alpaka = cms.untracked.PSet(
        backend = cms.untracked.string("serial_sync") # or cuda_async
    )
)

The validity of the explicitly chosen backend is checked at the time the configuration is processed at the worker node, and value that is incompatible with the node hardware (or process.options.accelerators) leads to an exception being thrown. It follows that this approach of defining Alpaka modules with explicitly selected backend does not really work with SwitchProducer (where all modules need to be constructed in the C++ side).

An alternative design that was considered was to encode the backend selection information in the C++ type field (e.g. after @alpaka). Expressing it in the configuration parameters instead allows the backend selection to be easily customized after the module definition (e.g. with a Modifier), and allows the selection logic to be extended with little technical constraints in the future (e.g. selected modules could use TBB when running on CPU).

The benefits of this model over SwitchProducer include

works also with SCRAM_ARCHs that don't support all accelerator backends (or anything beyond CPU)
works also for EDAnalyzer, EDFilter, and ESProducer in addition of EDProducer
reduces the duplication of module parameters (at least in simple cases)
allows all module member functions (including constructors) to use backend device specific functionality without any explicit checks about "if the backend is enabled"

Potential discussion points

The nomenclature for declaring "Alpaka modules" with @alpaka suffix
The nomenclature for setting the module's backend explicitly
Whether the backend selection PSet should be visible in the generated cfi file (and thus be visible for ConfDB parsing)
For what exactly the cfi files should be generated?
- This PR takes a perhaps too big step of generating cfi file only for FooProducer@alpaka. An alternative could be to generate cfi files also for serial_sync::FooProducer and cuda_async::FooProducer for some amount of time.
When to add the loading of HeterogeneousCore.AlpakaCore.ProcessAcceleratorAlpaka_cfi to Configuration.StandardSequences.Accelerators_cff

PR validation:

Unit tests run.

cmsbuild · 2023-01-19T01:32:51Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-40564/33797

This PR adds an extra 28KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File HeterogeneousCore/AlpakaCore/interface/alpaka/ESProducer.h modified in PR(s): Simplify Alpaka ESProducer model by making the data copies to device implicit #40403
- File HeterogeneousCore/AlpakaCore/interface/alpaka/ProducerBase.h modified in PR(s): Simplify Alpaka ESProducer model by making the data copies to device implicit #40403
- File HeterogeneousCore/AlpakaTest/plugins/TestAlpakaAnalyzer.cc modified in PR(s): Add support to multi layout portable collections (any number) #40285
- File HeterogeneousCore/AlpakaTest/plugins/alpaka/TestAlpakaGlobalProducer.cc modified in PR(s): Simplify Alpaka ESProducer model by making the data copies to device implicit #40403
- File HeterogeneousCore/AlpakaTest/test/testAlpakaModules.sh modified in PR(s): Simplify Alpaka ESProducer model by making the data copies to device implicit #40403

Code check has found code style and quality issues which could be resolved by applying following patch(s)

code-format:
https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-40564/33797/code-format.patch
e.g. curl -k https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-40564/33797/code-format.patch | patch -p1
You can also run scram build code-format to apply code format directly

Instead of generating a cfi file for each Alpaka backend for each module

cmsbuild · 2023-01-19T01:41:15Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-40564/33798

This PR adds an extra 28KB to repository

cmsbuild · 2023-01-19T01:41:39Z

A new Pull Request was created by @makortel (Matti Kortelainen) for master.

It involves the following packages:

FWCore/ParameterSet (core)
HeterogeneousCore/AlpakaCore (heterogeneous)
HeterogeneousCore/AlpakaTest (heterogeneous)

@cmsbuild, @smuzaffar, @Dr15Jones, @makortel, @fwyzard can you please review it and eventually sign? Thanks.
@missirol, @wddgit, @rovere this is something you requested to watch as well.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.

cms-bot commands are listed here

makortel · 2023-01-19T01:42:13Z

enable gpu

makortel · 2023-01-19T01:42:20Z

@cmsbuild, please test

FWCore/ParameterSet/src/defaultModuleLabel.cc

fwyzard · 2023-01-19T05:14:53Z

Personally I'd like something like

process.foo = cms.PortableEDProducer("FooProducer", ...

over

process.foo = cms.EDProducer("FooProducer@alpaka", ...

but it's just a matter of taste - I have not thought about any possible implications.

fwyzard · 2023-01-19T05:19:08Z

An alternative design that was considered was to encode the backend selection information in the C++ type field (e.g. after @alpaka).

Do you mean something like

process.foo = cms.EDProducer("FooProducer@alpaka@serial_sync", ...

?

But then isn't it simpler to just use the explicit module name,

process.foo = cms.EDProducer("alpaka_serial_sync::TestAlpakaProducer", ...

?

HeterogeneousCore/AlpakaTest/plugins/alpaka/TestAlpakaGlobalProducer.cc

HeterogeneousCore/AlpakaCore/python/ProcessAcceleratorAlpaka.py

makortel · 2023-01-31T22:20:43Z

Following the discussion in the HLT GPU developments meeting on Monday I updated the PR to add back the the cfi generation of the Alpaka modules with the explicit per-backend namespaces

This change has one consequence that is worth of noting: users calling ConfigurationDescriptions::add() in the fillDescriptions() must explicitly disambiguate the module labels for different backends.

fwyzard · 2023-01-31T22:23:31Z

+heterogeneous

makortel · 2023-01-31T22:24:29Z

+core

cmsbuild · 2023-01-31T22:24:54Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

makortel · 2023-01-31T22:25:57Z

@cmsbuild, please test

Probably better to refresh tests...

cmsbuild · 2023-02-01T04:53:13Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-708a0f/30301/summary.html
COMMIT: f93cd75
CMSSW: CMSSW_13_0_X_2023-01-31-1100/el8_amd64_gcc11
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/40564/30301/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 39 lines to the logs
Reco comparison results: 10 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3555495
DQMHistoTests: Total failures: 9
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3555464
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
Checked 211 log files, 162 edm output root files, 49 DQM output files
TriggerResults: no differences found

GPU Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 9 differences found in the comparisons
DQMHistoTests: Total files compared: 4
DQMHistoTests: Total histograms compared: 19862
DQMHistoTests: Total failures: 287
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 19575
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 3 files compared)
Checked 12 log files, 9 edm output root files, 4 DQM output files
TriggerResults: found differences in 2 / 3 workflows

rappoccio · 2023-02-01T07:52:33Z

+1

makortel added 2 commits January 12, 2023 21:19

Add ProcessAcceleratorAlpaka and AlpakaModuleTypeResolver

71c236b

Use collection size to ensure the expected backend is run

54d48f2

cmsbuild added this to the CMSSW_13_0_X milestone Jan 19, 2023

makortel mentioned this pull request Jan 19, 2023

Craft AlpakaModuleTypeResolver cms-sw/framework-team#450

Closed

cmsbuild added code-checks-pending core-pending heterogeneous-pending orp-pending pending-signatures tests-pending labels Jan 19, 2023

cmsbuild added code-checks-rejected and removed code-checks-pending labels Jan 19, 2023

Generate only one cfi file for each Alpaka module

48ef682

Instead of generating a cfi file for each Alpaka backend for each module

makortel force-pushed the alpakaModuleTypeResolver branch from c131538 to 48ef682 Compare January 19, 2023 01:36

cmsbuild added code-checks-pending and removed code-checks-rejected labels Jan 19, 2023

cmsbuild added code-checks-approved and removed code-checks-pending labels Jan 19, 2023

cmsbuild added tests-started and removed tests-pending labels Jan 19, 2023

makortel commented Jan 19, 2023

View reviewed changes

FWCore/ParameterSet/src/defaultModuleLabel.cc Show resolved Hide resolved

fwyzard reviewed Jan 19, 2023

View reviewed changes

HeterogeneousCore/AlpakaTest/plugins/alpaka/TestAlpakaGlobalProducer.cc Show resolved Hide resolved

fwyzard reviewed Jan 19, 2023

View reviewed changes

HeterogeneousCore/AlpakaCore/python/ProcessAcceleratorAlpaka.py Show resolved Hide resolved

cmsbuild added the tests-approved label Jan 25, 2023

cmsbuild added heterogeneous-approved and removed heterogeneous-pending labels Jan 31, 2023

cmsbuild added core-approved fully-signed and removed core-pending pending-signatures labels Jan 31, 2023

cmsbuild added tests-started and removed tests-approved labels Jan 31, 2023

cmsbuild added tests-approved and removed tests-started labels Feb 1, 2023

cmsbuild added orp-approved and removed orp-pending labels Feb 1, 2023

cmsbuild merged commit 7a9e56d into cms-sw:master Feb 1, 2023

makortel deleted the alpakaModuleTypeResolver branch February 1, 2023 14:16

makortel mentioned this pull request Feb 6, 2023

Follow-ups to "new Alpaka GPU framework" #40140

Open

20 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable "alternative module loading" approach for Alpaka modules #40564

Enable "alternative module loading" approach for Alpaka modules #40564

makortel commented Jan 19, 2023

cmsbuild commented Jan 19, 2023

cmsbuild commented Jan 19, 2023

cmsbuild commented Jan 19, 2023

makortel commented Jan 19, 2023

makortel commented Jan 19, 2023

fwyzard commented Jan 19, 2023

fwyzard commented Jan 19, 2023

makortel commented Jan 31, 2023

fwyzard commented Jan 31, 2023

makortel commented Jan 31, 2023

cmsbuild commented Jan 31, 2023

makortel commented Jan 31, 2023

cmsbuild commented Feb 1, 2023

rappoccio commented Feb 1, 2023

Enable "alternative module loading" approach for Alpaka modules #40564

Enable "alternative module loading" approach for Alpaka modules #40564

Conversation

makortel commented Jan 19, 2023

PR description:

Potential discussion points

PR validation:

cmsbuild commented Jan 19, 2023

cmsbuild commented Jan 19, 2023

cmsbuild commented Jan 19, 2023

makortel commented Jan 19, 2023

makortel commented Jan 19, 2023

fwyzard commented Jan 19, 2023

fwyzard commented Jan 19, 2023

makortel commented Jan 31, 2023

fwyzard commented Jan 31, 2023

makortel commented Jan 31, 2023

cmsbuild commented Jan 31, 2023

makortel commented Jan 31, 2023

cmsbuild commented Feb 1, 2023

Comparison Summary

GPU Comparison Summary

rappoccio commented Feb 1, 2023