model下面ocr_recognition使用attention_train_net加内存优化报错

[kolinwei]

单卡时没问题，多卡时报下面错误

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/work/weike/svn/baidu/paddle/test/cts_test/test_ocr_recognition.py", line 213, in test_train_ocr_recognition_attention_multi_gpu
    train_info = executor_network(model_name="attention", memory_optimize=True)
  File "/work/weike/svn/baidu/paddle/test/cts_test/test_ocr_recognition.py", line 105, in executor_network
    feed=get_feeder_data(data, place))
  File "/usr/local/lib/python2.7/dist-packages/paddle/fluid/parallel_executor.py", line 276, in run
    self.executor.run(fetch_list, fetch_var_name)
EnforceNotMet: Enforce failed. Expected end_idx <= dims_[0], but received end_idx:192 > dims_[0]:4.
The end row index is out of bound. at [/paddle/paddle/fluid/framework/tensor.cc:87]
PaddlePaddle Call Stacks:
0       0x7f6cbb433e86p paddle::platform::EnforceNotMet::EnforceNotMet(std::__exception_ptr::exception_ptr, char const*, int) + 486
1       0x7f6cbc4c9dcap paddle::framework::Tensor::Slice(int, int) const + 3578
2       0x7f6cbbe9e983p paddle::operators::ShrinkRNNMemoryOp::RunImpl(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) const + 979
3       0x7f6cbc41b64cp paddle::framework::OperatorBase::Run(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) + 252
4       0x7f6cbb4fbec9p paddle::framework::Executor::RunPreparedContext(paddle::framework::ExecutorPrepareContext*, paddle::framework::Scope*, bool, bool, bool) + 393
5       0x7f6cbb4fcca0p paddle::framework::Executor::Run(paddle::framework::ProgramDesc const&, paddle::framework::Scope*, int, bool, bool) + 128
6       0x7f6cbb9518c6p paddle::operators::WhileOp::RunImpl(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) const + 758
7       0x7f6cbc41b64cp paddle::framework::OperatorBase::Run(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) + 252
8       0x7f6cbc2607a7p
9       0x7f6cbc27e930p
10      0x7f6cbc27e195p paddle::framework::details::OpHandleBase::RunAndRecordEvent(std::function<void ()> const&) + 805
11      0x7f6cbc26027fp paddle::framework::details::ComputationOpHandle::RunImpl() + 95
12      0x7f6cbc27f235p paddle::framework::details::OpHandleBase::Run(bool) + 117
13      0x7f6cbc23ecbap
14      0x7f6cbc0bb6a3p std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, void> >::_M_invoke(std::_Any_data const&) + 35
15      0x7f6cbb5895d7p std::__future_base::_State_base::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>&, bool&) + 39
16      0x7f6d11110a99p
17      0x7f6cbc23dac2p
18      0x7f6cbb58b114p ThreadPool::ThreadPool(unsigned long)::{lambda()#1}::operator()() const + 404
19      0x7f6d0812e7e0p
20      0x7f6d111096bap
21      0x7f6d10e3f41dp clone + 109

[dzhwinter]

ShrinkRNNMemoryOp写错了

[kolinwei]

还有问题

[dzhwinter]

和@kolinwei确认过，单独跑test_ocr_recognition:attention_network_multigpu没有复现上述问题。原因是测试脚本中多个test network混用了一个scope。导致全局scope里存在的变量是未定义的，出现了上述问题。
正确使用方法如例子所示

Paddle/tools/test_runner.py

Line 34 in 340a104

with fluid.scope_guard(scope):