cuda-memcheck errors

[alexstrel]

cuda-memcheck utility returns CUDA_ERROR_INVALID_VALUE while the application executed successfully, probably a QMP-related issue. This is an example of the single-GPU execution with cuda-memcheck (code was built with QMP):
========= Program hit CUDA_ERROR_INVALID_VALUE (error 1) due to "invalid argument" on CUDA API call to cuPointerGetAttribute.
========= Saved host backtrace up to driver entry point at error
========= Host Frame:/usr/lib64/libcuda.so.1 (cuPointerGetAttribute + 0x174) [0x13d374]
========= Host Frame:./tests/invert_test_orig [0xb2b9d0]
========= Host Frame:./tests/invert_test_orig [0xca660d]
========= Host Frame:./tests/invert_test_orig [0xca6051]
========= Host Frame:./tests/invert_test_orig (mca_coll_self_allreduce_intra + 0x6f) [0xb601bf]
========= Host Frame:./tests/invert_test_orig [0xac009c]
========= Host Frame:./tests/invert_test_orig [0xaa54ab]
========= Host Frame:./tests/invert_test_orig [0x30e1be]
========= Host Frame:./tests/invert_test_orig [0x2ca6a2]
========= Host Frame:./tests/invert_test_orig [0x2d282e]
========= Host Frame:./tests/invert_test_orig [0x1e6123]
========= Host Frame:./tests/invert_test_orig [0x78062e]
========= Host Frame:./tests/invert_test_orig [0x7a401]
========= Host Frame:./tests/invert_test_orig [0x32a3f]
========= Host Frame:/lib64/libc.so.6 (__libc_start_main + 0xfd) [0x1ed1d]
========= Host Frame:./tests/invert_test_orig [0x315f1]

[mathiaswagner]

Some questions:

• Does this happen with quda 0.7 ? I guess invert_test_orig is the invert_test included in the 0.7 release?
• Did you try a build without QMP?
• ‘Executed successfully’ means no error message when run without cuda-memcheck ?

[alexstrel]

Yes, I used current muster (quda-0.7 release), and the invert_test application.
I'll check other options, i.e., pure MPI and pure single-GPU builds.
Yes , no errors without cuda-memcheck.

[mathiaswagner]

A single GPU build completes
cuda-memcheck ./invert_test
without errors for me (using CUDA 7.0).

I have not yet tried MPI or QMP.
It might also help to enable HOST_DEBUG for compilation and tracking down the location of the error.

[nmrcardoso]

Did you run your program with mpi? QMP was build with mpi? The CUDA-aware MPI env flag is active? I also got that kind of errors but only using mpi and if I set MV2_USE_CUDA in MVAPICH2. Also if I run a non CUDA-aware MPI program, with MV2_USE_CUDA active this gives a lot of that errors in cuda-memcheck, and off course with MV2_USE_CUDA=0, there is no cuda error.

[nmrcardoso]

I made a simple test only using cuPointerGetAttribute and passing a device and host pointers, no MPI here. if cuPointerGetAttribute is called and if if the pointer is not a device pointer
then cuda-memcheck always returns errors.
This is very annoying if we want to run cuda-memcheck and somewhere in the code there is a call to this function.

[mathiaswagner]

I have not checked in detail but does

http://docs.nvidia.com/cuda/cuda-driver-api/group__CUDA__UNIFIED.html#group__CUDA__UNIFIED_1g0c28ed0aff848042bc0533110e45820c

maybe explains the issue?

On Apr 22, 2015, at 14:20, nmrcardoso <[email protected]mailto:[email protected]> wrote:

I made a simple test only using cuPointerGetAttribute and passing a device and host pointers, no MPI here. if cuPointerGetAttribute is called and if if the pointer is not a device pointer
then cuda-memcheck always returns errors.
This is very annoying if we want to run cuda-memcheck and somewhere in the code there is a call to this function.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/222#issuecomment-95290359.

---

Mathias Wagner
Department of Physics SW 117 - Indiana University
Bloomington, IN 47405
email: [email protected]:[email protected]

[nmrcardoso]

cuda-memcheck errors from cuPointerGetAttribute are benign errors.
cuPointerGetAttribute is used to test whether the pointer is part of a cuda 'unified memory' or cuda managed memory object, however if the pointer passed is a "non cuda pointer" then cuda-memcheck triggers this as an error.
I think that there is no way to tell cuda-memcheck to ignore this kind of errors, just ignore it.