How to use multi GPU?

[HXACA]

How can I use multiple GPUs for training?

[XuyangBai]

Hi @HXACA Sorry we do not have the multiple GPU support.

[HXACA]

Thanks for your response.I try to use your default parameters to train the model with 3DMatch dataset，but the train_accuracy is always around 0.02. After 150 epoch，I stop the train and find the result is still very bad.

[XuyangBai]

Hi @HXACA I have just checked my code and find that the last commit causes this error. For the original implementation, I use contrastive loss with the hardest mining as the descriptor loss, and this setting is sensitive to the number of keypoint pairs used for calculating the loss, I suspect the current setting 256 might be too hard for the network to converge and I use 64 to get the result in the paper. You can change the keypts_num to 64 and I am pretty sure the network will work.

Another alternative is CircleLoss proposed in this paper. I find it shows better and fast convergence and has no convergence issue with keypts_num = 256 (which gives slightly better performance, and that's why I changed the keypts_num to 256 but I forget to change the loss type accordingly in that commit.)

I will add some notes for circle loss and double-check the configuration later. For now, you can try change the keypts_num to 64 training_3DMatch.py, or change the loss type to circle loss in KPFCNN_model.py. Sorry about the inconvenience.

Let me know if you still have this problem.

[HXACA]

Thanks for your response,I will try later.

[HXACA]

Sorry to bother, I had a new problem. When I tried the circle loss, after about 30 epoch, the loss suddenly increased to around 2, and the accuracy fell to single digits.. I tried to reduce the learning rate, but this problem still occurs.May I ask if you have encountered this problem？

[XuyangBai]

@HXACA what's your TensorFlow version and Cuda version? It seems very similar to the problem in here.

[HXACA]

I used RTX2080ti with CUDA10 and tf 1.12,I will try CUDA9 later.Thanks for your response.

[HXACA]

Sorry to bother again.I want know the keypts_num for KITTI dataset.Is the same with 3dMatch?I tried 256 keypts num and use circleloss,but not as good as your result.And I tried the default number 1024 but seem can't converge.

[XuyangBai]

@HXACA Hi, Could you try keypts_num = 64 and contrastive loss for KITTI? It seems I forgot to change the default value of keypts_num. I haven't tried circle loss for KITTI.

[ZYCheng-coder]

thanks fro your project @XuyangBai @HXACA
I found the same problem, and I've changed keypts_num to 64.
When train 14 epochs ， loss was closed to, the accuracy rate was close to 0.
I created environment by your file ,environment.yml, and RTX 2080ti.
could you give some advices about it?