Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multi-gpu failed #7

Open
yf1019 opened this issue Aug 3, 2023 · 1 comment
Open

Multi-gpu failed #7

yf1019 opened this issue Aug 3, 2023 · 1 comment

Comments

@yf1019
Copy link

yf1019 commented Aug 3, 2023

Hi, thanks to your great work.

I just trying to make this module into my own projects. While I found when I tried to use multi-gpu for training. The code will report errors as:

precompute_cuda.precompute(self.lbs_voxel_final, tfs, voxel_d, voxel_J, self.offset, self.scale)

RuntimeError: CUDA error: an illegal memory access was encountered

And I just check the source code "train.py" in this repo and I found it also reports an error when I try to use gpu > 2:

RuntimeError: Cowardly refusing to serialize non-leaf tensor which requires_grad, since autograd does not support crossing process boundaries. If you just want to transfer the data, call detach() on the tensor before serializing (e.g., putting it on the queue).

Anyone meets this error before?

@longxiang-ai
Copy link

Meeting the same error: "RuntimeError: CUDA error: an illegal memory access was encountered" when training on multi-gpus, have you solved this problem? @yf1019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants