scram b runtests fails to run alpaka GPU tests when USER_SCRAM_TARGET=auto

[fwyzard]

In a multi-targets development area, with USER_SCRAM_TARGET=auto, it looks like scram b runtests fails to run alpaka GPU tests.
This probably happens because the alpaka GPU tests are built only for the default target.

Set up with

cmsrel CMSSW_14_2_X_2024-09-01-0000
cd CMSSW_14_2_X_2024-09-01-0000
scram b enable-multi-targets
cmsenv

I get

$ USER_SCRAM_TARGET=default scram b runtests
>> Local Products Rules ..... started
>> Local Products Rules ..... done
>> Creating project symlinks
Creating test log file logs/el8_amd64_gcc12/testing.log
Pass    1s ... RecoTracker/PixelVertexFinding/deviceVertexFinderByDensity_tCudaAsync
Skip    0s ... RecoTracker/PixelVertexFinding/deviceVertexFinderByDensity_tROCmAsync (Failed to run rocmIsEnabled)
Pass    0s ... RecoTracker/PixelVertexFinding/deviceVertexFinderByDensity_tSerialSync
Pass    1s ... RecoTracker/PixelVertexFinding/deviceVertexFinderDBSCAN_tCudaAsync
Skip    0s ... RecoTracker/PixelVertexFinding/deviceVertexFinderDBSCAN_tROCmAsync (Failed to run rocmIsEnabled)
Pass    0s ... RecoTracker/PixelVertexFinding/deviceVertexFinderDBSCAN_tSerialSync
Pass    1s ... RecoTracker/PixelVertexFinding/deviceVertexFinderIterative_tCudaAsync
Skip    0s ... RecoTracker/PixelVertexFinding/deviceVertexFinderIterative_tROCmAsync (Failed to run rocmIsEnabled)
Pass    0s ... RecoTracker/PixelVertexFinding/deviceVertexFinderIterative_tSerialSync
Pass    1s ... RecoTracker/PixelVertexFinding/deviceVertexFinderOneKernel_tCudaAsync
Skip    0s ... RecoTracker/PixelVertexFinding/deviceVertexFinderOneKernel_tROCmAsync (Failed to run rocmIsEnabled)
Pass    1s ... RecoTracker/PixelVertexFinding/deviceVertexFinderOneKernel_tSerialSync

vs

$ USER_SCRAM_TARGET=auto scram b runtests
IMPORTANT: Setting CMSSW environment to use 'x86-64-v3' target.
>> Local Products Rules ..... started
>> Local Products Rules ..... done
>> Creating project symlinks
Creating test log file logs/el8_amd64_gcc12/testing.log
>>>> Running tests for scram_x86-64-v3 <<<<
Fail    0s ... RecoTracker/PixelVertexFinding/deviceVertexFinderByDensity_tCudaAsync
Skip    0s ... RecoTracker/PixelVertexFinding/deviceVertexFinderByDensity_tROCmAsync (Failed to run rocmIsEnabled)
Pass    1s ... RecoTracker/PixelVertexFinding/deviceVertexFinderByDensity_tSerialSync
Fail    0s ... RecoTracker/PixelVertexFinding/deviceVertexFinderDBSCAN_tCudaAsync
Skip    0s ... RecoTracker/PixelVertexFinding/deviceVertexFinderDBSCAN_tROCmAsync (Failed to run rocmIsEnabled)
Pass    0s ... RecoTracker/PixelVertexFinding/deviceVertexFinderDBSCAN_tSerialSync
Fail    0s ... RecoTracker/PixelVertexFinding/deviceVertexFinderIterative_tCudaAsync
Skip    0s ... RecoTracker/PixelVertexFinding/deviceVertexFinderIterative_tROCmAsync (Failed to run rocmIsEnabled)
Pass    0s ... RecoTracker/PixelVertexFinding/deviceVertexFinderIterative_tSerialSync
Fail    0s ... RecoTracker/PixelVertexFinding/deviceVertexFinderOneKernel_tCudaAsync
Skip    0s ... RecoTracker/PixelVertexFinding/deviceVertexFinderOneKernel_tROCmAsync (Failed to run rocmIsEnabled)
Pass    0s ... RecoTracker/PixelVertexFinding/deviceVertexFinderOneKernel_tSerialSync

[fwyzard]

assign core

[fwyzard]

@smuzaffar could you have a look ?

[cmsbuild]

New categories assigned: core

@Dr15Jones,@makortel,@smuzaffar you have been requested to review this Pull request/Issue and eventually sign? Thanks

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @fwyzard.

@Dr15Jones, @antoniovilela, @makortel, @mandrenguyen, @rappoccio, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[smuzaffar]

This probably happens because the alpaka GPU tests are built only for the default target.

@fwyzard , yes this is correct we had explicitly disabled building rocm/cuda backends for the extra micro-architecture. The problem is that scram is only adding $LOCALTOP/test/$SCRAM_ARCH/scram-<micro-arch> in to PATH when USER_SCRAM_TARGET=auto was used. I will fix this so that scram should add both $LOCALTOP/test/$SCRAM_ARCH/scram-<micro-arch> and $LOCALTOP/test/$SCRAM_ARCH so that it can find the default test executable

[smuzaffar]

cms-sw/cmsdist#9383 has been merged and it should fix this issue. I have opened backports for 14.1.X and 14.0.X IBs too

[fwyzard]

Thanks, tomorrow I'll test tonight's IB.

[fwyzard]

I confirm that in CMSSW_14_2_X_2024-09-02-2300 the alpaka/CUDA tests work using the auto and x86-64-v3 targets.

[smuzaffar]

+core

I guess we can close this issue now

[cmsbuild]

This issue is fully signed and ready to be closed.

[makortel]

@cmsbuild, please close