Reimplement the PortableCollection and related tests in CUDA [12.5.x] #39321

fwyzard · 2022-09-06T15:50:05Z

PR description:

To ease the migration from CUDA to Alpaka, reimplement in CUDA the PortableHostCollection and PortableDeviceCollection originally developed for the Alpaka data formats, along with the PortableTestObjects tests.

The CUDA version is in the cms::cuda namespace.

The data format functionality requires extending the cms::cuda::host::make_unique<T> to support pageable (i.e. non-pinned) host memory.

PR validation:

The new unit tests run successfully, both with and without GPUs:

$ cmsRun HeterogeneousCore/CUDATest/python/writer.py 
%MSG-i CUDAService:  (NoModuleName) 06-Sep-2022 17:39:19 CEST pre-events
CUDA runtime version 11.5, driver version 11.6, NVIDIA driver version 510.47.03
CUDA device 0: Tesla T4 (sm_75)
CUDA device 1: Tesla T4 (sm_75)
%MSG
...

$ edmDumpEventContent test.root 
Type                                  Module                 Label     Process    
----------------------------------------------------------------------------------
cms::cuda::PortableHostCollection<portabletest::TestSoALayout<128,false> >    "testProducer"         ""        "Writer"   
cms::cuda::PortableHostCollection<portabletest::TestSoALayout<128,false> >    "testProducerSerial"   ""        "Writer"   

$ CUDA_VISIBLE_DEVICES= cmsRun HeterogeneousCore/CUDATest/python/writer.py 
%MSG-i CUDAService:  (NoModuleName) 06-Sep-2022 17:39:58 CEST pre-events
CUDAService disabled by configuration
%MSG
...

$ edmDumpEventContent test.root 
Type                                  Module                 Label     Process    
----------------------------------------------------------------------------------
cms::cuda::PortableHostCollection<portabletest::TestSoALayout<128,false> >    "testProducer"         ""        "Writer"   
cms::cuda::PortableHostCollection<portabletest::TestSoALayout<128,false> >    "testProducerSerial"   ""        "Writer"   

$ cmsRun HeterogeneousCore/CUDATest/python/reader.py 
06-Sep-2022 17:40:26 CEST  Initiating request to open file file:test.root
06-Sep-2022 17:40:26 CEST  Successfully opened file file:test.root
...

Regarding the changes to the cms::cuda::host::unique_ptr<T>, they were tested also in CMSSW 12.4.8 on a GPU-enabled HLT workflow, to make sure they do not have any negative impact on the performance.

Vanilla CMSSW_12_4_8

Running 4 times over 10300 events with 8 jobs, each with 32 threads, 32 streams and 1 GPUs
   798.2 ±   0.2 ev/s (10000 events, 98.6% overlap)
   797.5 ±   0.2 ev/s (10000 events, 98.8% overlap)
   799.4 ±   0.2 ev/s (10000 events, 99.3% overlap)
   800.3 ±   0.2 ev/s (10000 events, 99.1% overlap)
 --------------------
   798.8 ±   1.3 ev/s

With the equivalent changes as in this PR:

Running 4 times over 10300 events with 8 jobs, each with 32 threads, 32 streams and 1 GPUs
   798.2 ±   0.2 ev/s (10000 events, 99.1% overlap)
   798.4 ±   0.2 ev/s (10000 events, 99.0% overlap)
   798.0 ±   0.2 ev/s (10000 events, 98.7% overlap)
   798.1 ±   0.2 ev/s (10000 events, 99.4% overlap)
 --------------------
   798.2 ±   0.2 ev/s

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

Backport of #39319 to CMSSW 12.5.x.

cmsbuild · 2022-09-06T15:50:31Z

A new Pull Request was created by @fwyzard (Andrea Bocci) for CMSSW_12_5_X.

It involves the following packages:

CUDADataFormats/Common (heterogeneous)
CUDADataFormats/PortableTestObjects (****)
DataFormats/Portable (heterogeneous)
DataFormats/PortableTestObjects (heterogeneous)
DataFormats/SoATemplate (heterogeneous)
HeterogeneousCore/AlpakaInterface (heterogeneous)
HeterogeneousCore/AlpakaTest (heterogeneous)
HeterogeneousCore/CUDATest (heterogeneous)
HeterogeneousCore/CUDAUtilities (heterogeneous)

The following packages do not have a category, yet:

CUDADataFormats/PortableTestObjects
Please create a PR for https://github.com/cms-sw/cms-bot/blob/master/categories_map.py to assign category

@cmsbuild, @makortel, @fwyzard can you please review it and eventually sign? Thanks.
@makortel, @rovere this is something you requested to watch as well.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.

cms-bot commands are listed here

Backported from Reimplement the PortableCollection and related tests in CUDA #39319

cmsbuild · 2022-09-06T16:01:21Z

Pull request #39321 was updated. @cmsbuild, @makortel, @fwyzard can you please check and sign again.

fwyzard · 2022-09-06T16:01:40Z

backport #39319

fwyzard · 2022-09-06T16:01:48Z

enable gpu

fwyzard · 2022-09-06T16:01:51Z

please test

cmsbuild · 2022-09-06T17:31:17Z

-1

Failed Tests: Build ClangBuild
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-8ecf42/27372/summary.html
COMMIT: c1f6600
CMSSW: CMSSW_12_5_X_2022-09-06-1100/el8_amd64_gcc10
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/39321/27372/install.sh to create a dev area with all the needed externals and cmssw changes.

Build

I found compilation error when building:

/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_5_X_2022-09-06-1100/src/HeterogeneousCore/AlpakaInterface/interface/CachedBufAlloc.h:76:42:   required from 'static alpaka::BufCpu cms::alpakatools::traits::CachedBufAlloc, void>::allocCachedBuf(const alpaka::DevCpu&, alpaka::QueueCudaRtNonBlocking, const TExtent&) [with TExtent = alpaka::Vec, unsigned int>; TElem = std::byte; TDim = std::integral_constant; TIdx = unsigned int; alpaka::QueueCudaRtNonBlocking = alpaka::uniform_cuda_hip::detail::QueueUniformCudaHipRt]'
/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_5_X_2022-09-06-1100/src/HeterogeneousCore/AlpakaInterface/interface/CachedBufAlloc.h:202:99:   required from 'auto cms::alpakatools::allocCachedBuf(const TDev&, TQueue, const TExtent&) [with TElem = std::byte; TIdx = unsigned int; TExtent = alpaka::Vec, unsigned int>; TQueue = alpaka::uniform_cuda_hip::detail::QueueUniformCudaHipRt; TDev = alpaka::DevCpu;  = void]'
/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_5_X_2022-09-06-1100/src/HeterogeneousCore/AlpakaInterface/interface/memory.h:117:58:   required from 'std::enable_if_t<((is_queue_v && is_unbounded_array_v) && (! is_array_v::type>)), typename cms::alpakatools::detail::buffer_type::type> cms::alpakatools::make_host_buffer(const TQueue&, alpaka_common::Extent) [with T = std::byte []; TQueue = alpaka::uniform_cuda_hip::detail::QueueUniformCudaHipRt; std::enable_if_t<((is_queue_v && is_unbounded_array_v) && (! is_array_v::type>)), typename cms::alpakatools::detail::buffer_type::type> = alpaka::BufCpu, unsigned int>; typename cms::alpakatools::detail::buffer_type::type = alpaka::BufCpu, unsigned int>; typename std::remove_extent<_Tp>::type = std::byte; alpaka_common::Extent = unsigned int]'
/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_5_X_2022-09-06-1100/src/DataFormats/Portable/interface/PortableHostCollection.h:36:64:   required from 'PortableHostCollection::PortableHostCollection(int32_t, const TQueue&) [with TQueue = alpaka::uniform_cuda_hip::detail::QueueUniformCudaHipRt;  = void; T = portabletest::TestSoALayout<>; int32_t = int]'
/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_5_X_2022-09-06-1100/src/HeterogeneousCore/AlpakaTest/plugins/alpaka/TestAlpakaTranscriber.cc:43:100:   required from here
/cvmfs/cms-ib.cern.ch/nweek-02749/el8_amd64_gcc10/external/alpaka/develop-20220811-f73576a1328c2c84ceb84f49e8371846/include/alpaka/mem/buf/Traits.hpp:140:86: error: incomplete type 'alpaka::trait::BufAllocMapped, std::integral_constant, std::byte, long unsigned int>' used in nested name specifier
  140 |         return trait::BufAllocMapped, TIdx, TDev>::allocMappedBuf(host, dev, extent);
      |                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~
gmake: *** [tmp/el8_amd64_gcc10/src/HeterogeneousCore/AlpakaTest/plugins/HeterogeneousCoreAlpakaTestPluginsPortableCudaAsync/alpaka/TestAlpakaTranscriber.cc.o] Error 1
>> Cuda Device Link tmp/el8_amd64_gcc10/src/HeterogeneousCore/AlpakaTest/plugins/HeterogeneousCoreAlpakaTestPluginsPortableCudaAsync/HeterogeneousCoreAlpakaTestPluginsPortableCudaAsync_cudadlink.o 
>> Building alpaka/cuda edm plugin tmp/el8_amd64_gcc10/src/HeterogeneousCore/AlpakaTest/plugins/HeterogeneousCoreAlpakaTestPluginsPortableCudaAsync/libHeterogeneousCoreAlpakaTestPluginsPortableCudaAsync.so

Clang Build

I found compilation error while trying to compile with clang. Command used:

USER_CUDA_FLAGS='--expt-relaxed-constexpr' USER_CXXFLAGS='-Wno-register -fsyntax-only' scram build -k -j 32 COMPILER='llvm compile'

>> Entering Package DataFormats/SoATemplate
>> Entering Package HeterogeneousCore/AlpakaTest
>> Entering Package HeterogeneousCore/CUDATest
>> Entering Package HeterogeneousCore/CUDAUtilities
>> Compile sequence completed for CMSSW CMSSW_12_5_X_2022-09-06-1100
gmake: *** [There are compilation/build errors. Please see the detail log above.] Error 1
+ eval scram build outputlog '&&' '(python3' /data/cmsbld/jenkins/workspace/ib-run-pr-tests/cms-bot/buildLogAnalyzer.py --logDir /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_5_X_2022-09-06-1100/tmp/el8_amd64_gcc10/cache/log/src '||' 'true)'
++ scram build outputlog
>> Entering Package CUDADataFormats/Common
Entering library rule at CUDADataFormats/Common
>> Compiling  /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_5_X_2022-09-06-1100/src/CUDADataFormats/Common/src/ProductBase.cc

fwyzard · 2022-09-06T17:57:11Z

please test with cms-sw/cmsdist#8069

cmsbuild · 2022-09-06T19:44:04Z

Pull request #39321 was updated. @cmsbuild, @makortel, @fwyzard can you please check and sign again.

cmsbuild · 2022-09-07T22:03:01Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-8ecf42/27409/summary.html
COMMIT: c9a50cb
CMSSW: CMSSW_12_5_X_2022-09-07-1100/el8_amd64_gcc10
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/39321/27409/install.sh to create a dev area with all the needed externals and cmssw changes.

GPU Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
Reco comparison had 3 failed jobs
DQMHistoTests: Total files compared: 4
DQMHistoTests: Total histograms compared: 19876
DQMHistoTests: Total failures: 8
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 19868
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 3 files compared)
Checked 12 log files, 9 edm output root files, 4 DQM output files
TriggerResults: no differences found

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 51
DQMHistoTests: Total histograms compared: 3695708
DQMHistoTests: Total failures: 2
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3695684
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 50 files compared)
Checked 212 log files, 49 edm output root files, 51 DQM output files
TriggerResults: no differences found

Move the test logic to a bash script to simplify the BuildFile.xml.

cmsbuild · 2022-09-07T22:36:49Z

Pull request #39321 was updated. @cmsbuild, @makortel, @fwyzard can you please check and sign again.

fwyzard · 2022-09-07T22:37:01Z

please test

fwyzard · 2022-09-07T22:39:23Z

+heterogeneous

cmsbuild · 2022-09-07T22:39:44Z

This pull request is fully signed and it will be integrated in one of the next CMSSW_12_5_X IBs after it passes the integration tests and once validation in the development release cycle CMSSW_12_6_X is complete. This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

cmsbuild · 2022-09-08T04:57:24Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-8ecf42/27418/summary.html
COMMIT: d2e962a
CMSSW: CMSSW_12_5_X_2022-09-07-2300/el8_amd64_gcc10
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/39321/27418/install.sh to create a dev area with all the needed externals and cmssw changes.

GPU Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
Reco comparison had 3 failed jobs
DQMHistoTests: Total files compared: 4
DQMHistoTests: Total histograms compared: 19876
DQMHistoTests: Total failures: 10
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 19866
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 3 files compared)
Checked 12 log files, 9 edm output root files, 4 DQM output files
TriggerResults: no differences found

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 4 differences found in the comparisons
DQMHistoTests: Total files compared: 51
DQMHistoTests: Total histograms compared: 3695708
DQMHistoTests: Total failures: 7
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 3695678
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -0.004 KiB( 50 files compared)
DQMHistoSizes: changed ( 312.0 ): -0.004 KiB MessageLogger/Warnings
Checked 212 log files, 49 edm output root files, 51 DQM output files
TriggerResults: no differences found

perrotta · 2022-09-11T20:52:08Z

+1

Add missing header

e1590fb

cmsbuild added this to the CMSSW_12_5_X milestone Sep 6, 2022

cmsbuild added heterogeneous-pending new-package-pending orp-pending pending-signatures tests-pending labels Sep 6, 2022

fwyzard force-pushed the cuda_backport_alpaka_tests branch from 30a5539 to c1f6600 Compare September 6, 2022 16:00

fwyzard changed the title ~~Cuda backport alpaka tests~~ Reimplement the PortableCollection and related tests in CUDA [12.5.x] Sep 6, 2022

cmsbuild added backport tests-started tests-rejected and removed tests-pending tests-started labels Sep 6, 2022

cmsbuild added requires-external tests-started and removed tests-rejected labels Sep 6, 2022

fwyzard force-pushed the cuda_backport_alpaka_tests branch from c1f6600 to e84b4e3 Compare September 6, 2022 19:43

cmsbuild added tests-pending and removed tests-started requires-external labels Sep 6, 2022

Add support for pageable host memory

33f26c9

cmsbuild added tests-started and removed tests-pending new-package-pending labels Sep 7, 2022

cmsbuild added tests-approved and removed tests-started labels Sep 7, 2022

Run similar tests in different directories to avoid conflicts

d2e962a

Move the test logic to a bash script to simplify the BuildFile.xml.

fwyzard force-pushed the cuda_backport_alpaka_tests branch from c9a50cb to d2e962a Compare September 7, 2022 22:36

cmsbuild added tests-pending and removed tests-approved labels Sep 7, 2022

cmsbuild added tests-started and removed tests-pending labels Sep 7, 2022

cmsbuild added fully-signed heterogeneous-approved and removed pending-signatures heterogeneous-pending labels Sep 7, 2022

cmsbuild added tests-approved backport-ok and removed tests-started backport labels Sep 8, 2022

cmsbuild added orp-approved and removed orp-pending labels Sep 11, 2022

cmsbuild merged commit 75fe478 into cms-sw:CMSSW_12_5_X Sep 11, 2022

fwyzard deleted the cuda_backport_alpaka_tests branch September 29, 2022 09:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reimplement the PortableCollection and related tests in CUDA [12.5.x] #39321

Reimplement the PortableCollection and related tests in CUDA [12.5.x] #39321

fwyzard commented Sep 6, 2022

cmsbuild commented Sep 6, 2022 •

edited

Loading

cmsbuild commented Sep 6, 2022

fwyzard commented Sep 6, 2022

fwyzard commented Sep 6, 2022

fwyzard commented Sep 6, 2022

cmsbuild commented Sep 6, 2022

fwyzard commented Sep 6, 2022

cmsbuild commented Sep 6, 2022

cmsbuild commented Sep 7, 2022

cmsbuild commented Sep 7, 2022

fwyzard commented Sep 7, 2022

fwyzard commented Sep 7, 2022

cmsbuild commented Sep 7, 2022

cmsbuild commented Sep 8, 2022

perrotta commented Sep 11, 2022

Reimplement the PortableCollection and related tests in CUDA [12.5.x] #39321

Reimplement the PortableCollection and related tests in CUDA [12.5.x] #39321

Conversation

fwyzard commented Sep 6, 2022

PR description:

PR validation:

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

cmsbuild commented Sep 6, 2022 • edited Loading

cmsbuild commented Sep 6, 2022

fwyzard commented Sep 6, 2022

fwyzard commented Sep 6, 2022

fwyzard commented Sep 6, 2022

cmsbuild commented Sep 6, 2022

Build

Clang Build

fwyzard commented Sep 6, 2022

cmsbuild commented Sep 6, 2022

cmsbuild commented Sep 7, 2022

GPU Comparison Summary

Comparison Summary

cmsbuild commented Sep 7, 2022

fwyzard commented Sep 7, 2022

fwyzard commented Sep 7, 2022

cmsbuild commented Sep 7, 2022

cmsbuild commented Sep 8, 2022

GPU Comparison Summary

Comparison Summary

perrotta commented Sep 11, 2022

cmsbuild commented Sep 6, 2022 •

edited

Loading