Prediction with py_paddle and 'use_gpu=1'

[CrossLee1]

I use the py_paddle for prediction.
If I use this command swig_paddle.initPaddle("--use_gpu=0"), everything works find.
But when I change --use_gpu=0 to --use_gpu=1, error occurs.

I1103 13:27:47.719333 4879 Util.cpp:155] commandline: --use_gpu=1
I1103 13:27:50.586648 4879 Util.cpp:130] Calling runInitFunctions
I1103 13:27:50.587080 4879 Util.cpp:143] Call runInitFunctions done.
[WARNING 2016-11-03 13:27:50,601 default_decorators.py:40] please use keyword arguments in paddle config.
I1103 13:27:50.623359 4879 GradientMachine.cpp:123] Loading parameters from ./models/pass-00007/
F1103 13:27:51.092999 4879 Matrix.cpp:475] Not supported
*** Check failure stack trace: ***
@ 0x7fc2ad19d6b8 google::LogMessage::Fail()
@ 0x7fc2ad19d614 google::LogMessage::SendToLog()
@ 0x7fc2ad19cf98 google::LogMessage::Flush()
@ 0x7fc2ad1a003f google::LogMessageFatal::~LogMessageFatal()
@ 0x7fc2acf92c97 paddle::GpuMatrix::mul()
@ 0x7fc2acf1907a paddle::FullMatrixProjection::forward()
@ 0x7fc2acf07766 paddle::MixedLayer::forward()
@ 0x7fc2ace72800 paddle::NeuralNetwork::forward()
@ 0x7fc2ace07cf9 _wrap_GradientMachine_forward
@ 0x7fc2b49645f3 PyEval_EvalFrameEx
@ 0x7fc2b4966685 PyEval_EvalCodeEx
@ 0x7fc2b4962f97 PyEval_EvalFrameEx
@ 0x7fc2b4966685 PyEval_EvalCodeEx
@ 0x7fc2b4962f97 PyEval_EvalFrameEx
@ 0x7fc2b49630c8 PyEval_EvalFrameEx
@ 0x7fc2b4966685 PyEval_EvalCodeEx
@ 0x7fc2b49667bc PyEval_EvalCode
@ 0x7fc2b4990039 PyRun_FileExFlags
@ 0x7fc2b49914a1 PyRun_SimpleFileExFlags
@ 0x7fc2b49a7fa1 Py_Main
@ 0x7fc2b3b9bbd5 __libc_start_main
@ 0x4008d9 (unknown)
@ (nil) (unknown)
Aborted

What is the problem? Thanks~

[luotao1]

You can paste the 475 line of your Matrix.cpp, maybe this function is not supported for gpu now.

[CrossLee1]

void GpuMatrix::mul(const MatrixPtr a, const MatrixPtr b, real scaleAB,
real scaleT) {
GpuMatrixPtr a_ptr = std::dynamic_pointer_cast(a);
GpuMatrixPtr b_ptr = std::dynamic_pointer_cast(b);
GpuSparseMatrixPtr a_ptr_s = std::dynamic_pointer_cast(a);
GpuSparseMatrixPtr b_ptr_s = std::dynamic_pointer_cast(b);

if (a_ptr && b_ptr) {
mul(_a_ptr, b_ptr, scaleAB, scaleT);
} else if (a_ptr_s && b_ptr) {
mul(_a_ptr_s, _b_ptr, scaleAB, scaleT);
} else if (a_ptr && b_ptr_s) {
mul(_a_ptr, _b_ptr_s, scaleAB, scaleT);
} else {
_LOG(FATAL) << "Not supported";
}
}

It seems to be the multiplication of matrices by GPU.

[luotao1]

It seems that run the last branch (a_ptr_s && b_ptr_s), but it not supported now. Is your multiplication on sparse-matrix * sparse-matrix?

[CrossLee1]

It seems so.
The ReluActivation may cause sparse-matrix * sparse-matrix ...

[luotao1]

@hedaoyuan when we will support sparse-matrix * sparse-matrix in GPU?

[hedaoyuan]

What kind of model prediction need sparse mul sparse and return a dense matrix?
May be a bug, should not run to here.

[CrossLee1]

@hedaoyuan My network for test is just as follows:

Layer(
  type = 'fc',
  name = 'up',
  size = up_size,
  active_type = 'relu',
  inputs = [Input('input')],
  bias = True
)

Layer(
  type = 'fc',
  name = 'hidden',
  size = hidden_size,
  active_type = 'tanh',
  inputs = [Input('up')],
  bias = True
)

Layer(
  type = 'fc',
  name = 'output',
  size = label_size,
  active_type = 'sigmoid',
  inputs = [Input('hidden')],
  bias = True
)

Outputs('output')

[emailweixu]

@CrossLee1 your network does not match the error message. The error messages shows that the error is in MixedLayer, but your network config does not have MixedLayer.

[CrossLee1]

@emailweixu but that is exactly what I use for training(exclude the loss layer) and testing.

[reyoung]

@CrossLee1 Please make sure what network config you used.

Just add a print line could be cool

print("I'm using this config file")

[emailweixu]

reproduced. @wangyang59 is looking at it.

[wangyang59]

@CrossLee1
I have reproduced your problem. Could you please share your code using py_paddle api? I suspect that you create the input data using createCpuDenseFromNumpy.

[CrossLee1]

@wangyang59
The following is part of my prediction code using py_paddle.
Thanks for the patience of all of your guys :)

swig_paddle.initPaddle("--use_gpu=1")
conf = parse_config("./conf_test.py", "")
network = swig_paddle.GradientMachine.createFromConfigProto(conf.model_config)
network.loadParameters("./models/pass-00011/")

slots = [DenseSlot(1024)]
converter = util.DataProviderWrapperConverter(True, slots)
.........
# feat_encode's type is numpy, convert it to list of list
feat_list = []
for row in range(feat_decode.shape[0]):
    feat_list.append(feat_decode[row,:].tolist())
inArg = converter([[feat_list]])
output = network.forwardTest(inArg)[0]
........

`

[wangyang59]

Hi @reyoung @emailweixu
I think I found the problem. In py_paddle/util.py, the function call of swig_paddle.Matrix.createDense set useGpu=False as default, so the input arguments are in CPU mode which causes the error.

class DataProviderWrapperConverter(object): (line 263)
class DenseValueConverter(object):
def call(self, slot_idx, arg):
mat = swig_paddle.Matrix.createDense(self.buf,
len(self.buf) / self.dim,
self.dim)
arg.setSlotValue(slot_idx, mat)

[CrossLee1]

@wangyang59

How to solve this error then?

[emailweixu]

fixed by #857

[hedaoyuan]

fixed by #1482