GPU使用multiBinaryLabelCrossEntropy问题

[CrossLee1]

我在GPU下训练，使用multiBinaryLabelCrossEntropy的cost，但是报下面的错误，请问是什么问题呢？

F0426 10:41:39.498152 20984 hl_cuda_device.cc:566] Check failed: cudaSuccess == cudaStat (0 vs. 77) Cuda Error: an illegal memory access was encountered
*** Check failure stack trace: ***
@ 0xff34a0 google::LogMessage::Fail()
@ 0xff33fc google::LogMessage::SendToLog()
@ 0xff2d80 google::LogMessage::Flush()
@ 0xff5e27 google::LogMessageFatal::~LogMessageFatal()
@ 0xfb5375 hl_stream_synchronize()
@ 0xfb9f6f hl_matrix_multi_binary_cross_entropy()
@ 0xd25592 paddle::GpuMatrix::multiBinaryLabelCrossEntropy()
@ 0xae957b paddle::MultiBinaryLabelCrossEntropy::forwardImp()
@ 0xae5d24 paddle::CostLayer::forward()
@ 0xbc37a1 paddle::NeuralNetwork::forward()
@ 0xbad6f8 paddle::TrainerThread::forward()
@ 0xbad3ae paddle::TrainerThread::computeThread()
@ 0xbacee1 _ZZN6paddle13TrainerThread5startEvENKUlvE_clEv
@ 0xbb30f0 _ZNSt12_Bind_simpleIFZN6paddle13TrainerThread5startEvEUlvE_vEE9_M_invokeIIEEEvSt12_Index_tupleIIXspT_EEE
@ 0xbb2e4f _ZNSt12_Bind_simpleIFZN6paddle13TrainerThread5startEvEUlvE_vEEclEv
@ 0xbb2cb0 _ZNSt6thread5_ImplISt12_Bind_simpleIFZN6paddle13TrainerThread5startEvEUlvE_vEEE6_M_runEv
@ 0x7fe6666b52d0 execute_native_thread_routine
@ 0x318b207851 (unknown)
@ 0x318aee767d (unknown)
@ (nil) (unknown)

[hedaoyuan]

I will check this function. And is there any way to reproduce the problem.

[hedaoyuan]

This illegal memory access problem may be due to incorrect input. @CrossLee1 #1907 add the inspection of input data, and if (row_col[i] >= dimN) will print like

Error: col_num=1001, i=1, row_col[i]=1072468318
F0426 21:08:35.725378 12051 hl_cuda_device.cc:566] Check failed: cudaSuccess == cudaStat (0 vs. 77) Cuda Error: an illegal memory access was encountered
``