Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory leak #45

Open
MathisJia opened this issue Dec 27, 2021 · 3 comments
Open

Memory leak #45

MathisJia opened this issue Dec 27, 2021 · 3 comments

Comments

@MathisJia
Copy link

作者你好,我在训练PVEN模型的过程中发现有内存泄漏的问题,每次运行 eval_ 函数后会有额外的内存占用。我目前找到的可能的问题是Clck_R1_mAP 类中调用 compute 时产生的线程池未停止运行。想请您再确认一下

@silverbulletmdc
Copy link
Owner

是显存还是内存?多线程应该不会有问题,也不涉及到gpu计算。eval时确实会带来额外的显存占用。但是我的gpu显存足够大,所以没有出现类似问题。。

@MathisJia
Copy link
Author

是内存问题 , 在计算mAP时调用了 eval_func_mp 这个函数,这里面创建了线程池,但函数结束时没有释放掉,所以在训练过程中每次计算当前mAP时(每10个epoch)都会创建新的进程池而且不会被kill,所以导致了内存泄漏。 没有涉及到显存。

@silverbulletmdc
Copy link
Owner

可能是会有问题,我之前默认函数退出时相关的回收操作会在Pool的析构函数里执行。你把创建pool的操作改成with Pool() as pool:应该就可以了。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants