[LTO] Unit test testCudaCheck failing for aarch64 architecture

[aandvalenzuela]

Hello,

Test testCudaCheck (module HeterogeneousCore/CUDAUtilities) is failing since 1st of Feb in LTO IBs (aarch64 architecture only) due to segmentation violation when checking the driver API:

/data/cmsbld/jenkins_a/workspace/build-any-ib/w/tmp/BUILDROOT/d82d494993e018306d251480a039b1b4/opt/cmssw/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_LTO_X_2023-02-19-0000/src/HeterogeneousCore/CUDAUtilities/test/testCudaCheck.cpp:13: FAILED:
  {Unknown expression after the reported line}
due to a fatal error condition:
  SIGSEGV - Segmentation violation signal

===============================================================================
test cases: 1 | 1 failed
assertions: 2 | 1 passed | 1 failed


 *** Break *** segmentation violation

See stacktrace.

Test was added on 1st of Feb via #40619 and it succeeds since then in amd64, but not on aarch64. Tests are supposed to pass on machines with and without GPUs (#40619 (comment)). I can reproduce the issue in all our aarch64 machines.

Thanks,
Andrea.

[cmsbuild]

A new Issue was created by @aandvalenzuela Andrea Valenzuela.

@Dr15Jones, @perrotta, @dpiparo, @rappoccio, @makortel, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[aandvalenzuela]

assign heterogeneous

[cmsbuild]

New categories assigned: heterogeneous

@fwyzard,@makortel you have been requested to review this Pull request/Issue and eventually sign? Thanks

[aandvalenzuela]

Not sure why it succeeds in non-LTO IBs:

===== Test "testCudaCheck" ====
===============================================================================
All tests passed (4 assertions in 1 test case)


---> test testCudaCheck succeeded

CMSSW_13_1_X_2023-02-20-2300 for el8_aarch64_gcc11 also.

[makortel]

I took a look and it seems that the error and message in

cmssw/HeterogeneousCore/CUDAUtilities/interface/cudaCheck.h

Lines 45 to 49 in 918a7fc

	const char* error;
	const char* message;
	cuGetErrorName(result, &error);
	cuGetErrorString(result, &message);
	abortOnCudaError(file, line, cmd, error, message, description);

stay uninitialized after the cuGetErrorName() and cuGetErrorString() calls. Ok, I didn't really check that, but their value in the abortOnCudaError() are garbage, and initializing them to nullptr seems to fix the crash.

[makortel]

Fixed in #40840

[aandvalenzuela]

Thanks @makortel!