ValueError: an illegal memory access was encountered terminate called after throwing an instance of 'std::runtime_error' what(): ./include/cudarray/common.hpp:95: an illegal memory access was encountered 中止 (コアダンプ)

[oscarriddle]

Hi Andersbll,

GTX1080 with CUDA8.0.27 and cudnn 5.1 here.
Installed cudarray with cuda back-end. Then try running this project but failed because illegal memory access.
LOG:

---

Traceback (most recent call last):
File "neural_artistic_style.py", line 138, in
run()
File "neural_artistic_style.py", line 130, in run
cost = np.mean(net.update())
File "/home/caoxiaolun/Workspace/CNN/neural_artistic_style/neural_artistic_style-master/style_network.py", line 150, in update
weight = float(self.style_weights[l]) / norm
File "/usr/local/lib/python2.7/dist-packages/cudarray-0.1.dev0-py2.7-linux-x86_64.egg/cudarray/cudarray.py", line 124, in rdiv
return elementwise.divide(other, self)
File "/usr/local/lib/python2.7/dist-packages/cudarray-0.1.dev0-py2.7-linux-x86_64.egg/cudarray/elementwise.py", line 167, in divide
return binary(elementwise.div_op, x1, x2, out)
File "/usr/local/lib/python2.7/dist-packages/cudarray-0.1.dev0-py2.7-linux-x86_64.egg/cudarray/elementwise.py", line 87, in binary
out = cudarray.empty(array.shape, dtype=out_dtype)
File "/usr/local/lib/python2.7/dist-packages/cudarray-0.1.dev0-py2.7-linux-x86_64.egg/cudarray/cudarray.py", line 246, in empty
return ndarray(shape, dtype=dtype)
File "/usr/local/lib/python2.7/dist-packages/cudarray-0.1.dev0-py2.7-linux-x86_64.egg/cudarray/cudarray.py", line 36, in init
self._data = ArrayData(self.size, dtype, np_data)
File "cudarray/wrap/array_data.pyx", line 16, in cudarray.wrap.array_data.ArrayData.init (./cudarray/wrap/array_data.cpp:1473)
File "cudarray/wrap/cudart.pyx", line 12, in cudarray.wrap.cudart.cudaCheck (./cudarray/wrap/cudart.cpp:824)
ValueError: an illegal memory access was encountered
terminate called after throwing an instance of 'std::runtime_error'
what(): ./include/cudarray/common.hpp:95: an illegal memory access was encountered
中止 (コアダンプ)

---

Besides, if I switch user to root and run, CUDA back-end will be unavailable: "CUDArray: CUDA back-end not available, using NumPy." But normal user will not show up this.

Thanks for your help.
oscarriddle

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 367.44 Driver Version: 367.44 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 1080 Off | 0000:01:00.0 On | N/A |
| 0% 40C P8 10W / 210W | 345MiB / 8112MiB | 1% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 935 G /usr/lib/xorg/Xorg 231MiB |
| 0 1558 G compiz 109MiB |
| 0 1894 G /usr/lib/firefox/firefox 2MiB |
+-----------------------------------------------------------------------------+

[oscarriddle]

Hi,

In previous issue: here
It seems that this issue is somehow because of out of memory. I checked nvidia-smi and find at peak, python will consume more than 2GB memory.

I just tried to reproduce the image by command:
python neural_artistic_style.py --subject images/tuebingen.jpg --style images/starry_night.jpg
Not sure why there will be an out of memory issue.

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 367.44 Driver Version: 367.44 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 1080 Off | 0000:01:00.0 On | N/A |
| 72% 55C P2 191W / 210W | 2800MiB / 8112MiB | 96% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 935 G /usr/lib/xorg/Xorg 279MiB |
| 0 1558 G compiz 119MiB |
| 0 1894 G /usr/lib/firefox/firefox 2MiB |
| 0 15425 C python 2395MiB |
+-----------------------------------------------------------------------------+

[oscarriddle]

Continue. I tried to step by step check whether I lost something so now cudarray cannot work.

1. By running CUDA 8.0RC samples, the CUDA 8.0RC is working. check
2. cuDNN is installed correctly. check
3. After including export path and env variable CUDNN_ENABLED=1 into /etc/profile. Now cudarray is installed and can be imported. check
4. I tried cudarray examples/test.py then got below log:

---

$python test.py
True
.
.
.
True
True
True
False
True
True
True
False
(3.0232945639326929, 3.0016344)
(10.010378811733982, 9.9944296)
(0.49997663711219476, 0.50012529)
(0.28865082930213165, 0.28864652)
(10.019171134185873, 10.004223)
(11.540598502938147, 11.549608)
Traceback (most recent call last):
File "test.py", line 494, in
run()
File "test.py", line 488, in run
test_reduce()
File "test.py", line 347, in test_reduce
print(np.allclose(c_np, np.array(c_ca)))
File "/usr/local/lib/python2.7/dist-packages/cudarray-0.1.dev0-py2.7-linux-x86_64.egg/cudarray/cudarray.py", line 42, in array
self._data.to_numpy(np_array)
File "cudarray/wrap/array_data.pyx", line 24, in cudarray.wrap.array_data.ArrayData.to_numpy (./cudarray/wrap/array_data.cpp:1593)
File "cudarray/wrap/cudart.pyx", line 12, in cudarray.wrap.cudart.cudaCheck (./cudarray/wrap/cudart.cpp:824)
ValueError: an illegal memory access was encountered
terminate called after throwing an instance of 'std::runtime_error'
what(): ./include/cudarray/common.hpp:95: an illegal memory access was encountered
中止 (コアダンプ)

---

This also showd that the cudarray is not working. Not sure whether because it cannot utilize the cuDNN and CUDA, so any memory allocation(even very tiny) will exceed the limitation. Still not sure how to solve it.

[atonn]

just wanted to note that I'm having the same issue on a GTX1080, which is funny because it worked a month or so ago with a different GTX1080. Other deep learning frameworks are working fine right now, will post again once I find out more, probably will go with a system reinstall first

[nudtfuruigang]

have the same issue on a GTX850M, which makes me disappointed

[wlxyhy]

the same problem, is there anyone know how to solve it?

[albertyou2]

@andersbll
@oscarriddle
I also met this problem.here is the log:

Traceback (most recent call last):
File "neural_artistic_style.py", line 138, in
run()
File "neural_artistic_style.py", line 130, in run
cost = np.mean(net.update())
File "/home/sojoyoo/neutral_style_adv_2/style_network.py", line 150, in update
weight = float(self.style_weights[l]) / norm
File "/home/sojoyoo/anaconda2/lib/python2.7/site-packages/cudarray-0.1.dev0-py2.7-linux-x86_64.egg/cudarray/cudarray.py", line 124, in rdiv
return elementwise.divide(other, self)
File "/home/sojoyoo/anaconda2/lib/python2.7/site-packages/cudarray-0.1.dev0-py2.7-linux-x86_64.egg/cudarray/elementwise.py", line 167, in divide
return binary(elementwise.div_op, x1, x2, out)
File "/home/sojoyoo/anaconda2/lib/python2.7/site-packages/cudarray-0.1.dev0-py2.7-linux-x86_64.egg/cudarray/elementwise.py", line 87, in binary
out = cudarray.empty(array.shape, dtype=out_dtype)
File "/home/sojoyoo/anaconda2/lib/python2.7/site-packages/cudarray-0.1.dev0-py2.7-linux-x86_64.egg/cudarray/cudarray.py", line 246, in empty
return ndarray(shape, dtype=dtype)
File "/home/sojoyoo/anaconda2/lib/python2.7/site-packages/cudarray-0.1.dev0-py2.7-linux-x86_64.egg/cudarray/cudarray.py", line 36, in init
self._data = ArrayData(self.size, dtype, np_data)
File "cudarray/wrap/array_data.pyx", line 16, in cudarray.wrap.array_data.ArrayData.init (./cudarray/wrap/array_data.cpp:1465)
File "cudarray/wrap/cudart.pyx", line 12, in cudarray.wrap.cudart.cudaCheck (./cudarray/wrap/cudart.cpp:816)
ValueError: an illegal memory access was encountered
terminate called after throwing an instance of 'std::runtime_error'
what(): ./include/cudarray/common.hpp:95: an illegal memory access was encountered
已放弃 (核心已转储)

env is ： Ubuntu 16.04 cuda 8.0,cudnn 5.0, gforce gtx 1070(8G memory)