RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED

[GuangZhaoZhou]

when I run "python train.py --config-file experiments/afm_atrous.yaml --gpu 0"
it shows that "Constructing DeepLabv3+ model...
Number of output channels: 2
Output stride: 16
Number of Input Channels: 3

Training AT epoch = 1
current learning rate = [0.01]

/home/zgz/anaconda3/envs/afm/lib/python3.5/site-packages/torch/nn/functional.py:2457: UserWarning: nn.functional.upsample is deprecated. Use nn.functional.interpolate instead.
warnings.warn("nn.functional.upsample is deprecated. Use nn.functional.interpolate instead.")
Traceback (most recent call last):
File "train.py", line 38, in
system.train(cfg, args.epoch)
File "/home/zgz/Documents/catkin_ws/cvpr2019_line_detector/afm_cvpr2019/modeling/afm.py", line 117, in train
avgLoss = step(epoch)
File "/home/zgz/Documents/catkin_ws/cvpr2019_line_detector/afm_cvpr2019/modeling/afm.py", line 94, in step
loss.backward()
File "/home/zgz/anaconda3/envs/afm/lib/python3.5/site-packages/torch/tensor.py", line 107, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "/home/zgz/anaconda3/envs/afm/lib/python3.5/site-packages/torch/autograd/init.py", line 93, in backward
allow_unreachable=True) # allow_unreachable flag
RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED
"
why ?
I appreciate your favor