发票数据集训练报错：Out of memory error on GPU 0. Cannot allocate 14.406982MB memory on GPU 0, 10.746094GB memory has been allocated and available memory is only 15.562500MB.

[dizhenx]

下载的官方的发票数据集做训练。运行python tools/train.py -c ./fapiao/train_data/ser_vi_layoutxlm.yml -o Global.save_model_dir=./output/kie/ 报以下错误。batchsize和num_works都调为1了还是报错。单卡显存有11G，无其他程序占用。
Error Message Summary:

ResourceExhaustedError:

Out of memory error on GPU 0. Cannot allocate 14.406982MB memory on GPU 0, 10.746094GB memory has been allocated and available memory is only 15.562500MB.

Please check whether there is any other process using GPU 0.

1. If yes, please stop them, or start PaddlePaddle on another GPU.
2. If no, please decrease the batch size of your model.
   If the above ways do not solve the out of memory problem, you can try to use CUDA managed memory. The command is export FLAGS_use_cuda_managed_memory=false.
   (at /paddle/paddle/fluid/memory/allocation/cuda_allocator.cc:95)
   为啥只能使用1个gpu，不能设置多卡训练吗？

[livingbody]

• 可以多卡训练，用paddle.distributed.launch --gpus '0,1,2,3' 。。。
• 参考

# 单机多卡训练，通过 --gpus 参数设置使用的GPU ID
python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/det/det_mv3_db.yml \
     -o Global.pretrained_model=./pretrain_models/MobileNetV3_large_x0_5_pretrained

• 链接https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.6/doc/doc_ch/detection.md

[shiyutang]

@livingbody 需要进一步实验看是否是就bs=1依旧超过11g显存

[dizhenx]

是的

[livingbody]

下载的官方的发票数据集做训练。运行python tools/train.py -c ./fapiao/train_data/ser_vi_layoutxlm.yml -o Global.save_model_dir=./output/kie/ 报以下错误。batchsize和num_works都调为1了还是报错。单卡显存有11G，无其他程序占用。

Error Message Summary:
ResourceExhaustedError:

Out of memory error on GPU 0. Cannot allocate 14.406982MB memory on GPU 0, 10.746094GB memory has been allocated and available memory is only 15.562500MB.

Please check whether there is any other process using GPU 0.

1. If yes, please stop them, or start PaddlePaddle on another GPU.
2. If no, please decrease the batch size of your model.
   If the above ways do not solve the out of memory problem, you can try to use CUDA managed memory. The command is export FLAGS_use_cuda_managed_memory=false.
   (at /paddle/paddle/fluid/memory/allocation/cuda_allocator.cc:95)
   为啥只能使用1个gpu，不能设置多卡训练吗？

能否在aistudio上分享项目给我，我查看下具体情况。

[livingbody]

是的

联系我微信：livingbody

[adamzhg]

@livingbody @dizhenx 请问问题解决了吗？
我这GPU是单卡，显存只有8G，也报类似的问题，是参考https://aistudio.baidu.com/projectdetail/4823162做的发票数据训练。

[ericyeyeye]

@livingbody @dizhenx 请问问题解决了吗？ 我这GPU是单卡，显存只有8G，也报类似的问题，是参考https://aistudio.baidu.com/projectdetail/4823162做的发票数据训练。

可以提供訓練的yml檔，才能瞭解你的設定，det model 中Eval batch_size_per_card只能設為1。

[papersuper]

请问最后解决了吗

[UserWangZz]

该issue长时间未更新，暂将此issue关闭，如有需要可重新开启。