Slow training process

[XuyangBai]

Hi @chrischoy Thanks for your sharing. I tried your code on 3DMatch dataset using the default configuration and found the training process is very slow. Specifically it took about one and a half hour for one epoch. (as you mentioned in the paper, you trained FCGF for 100 epochs, which means more than one week in my configuration). The GPU memory it took is only less than 5000 MB and GPU utility is less than 10% but CPU utility is high. I wonder is it normal situation and what's the most time-consuming part ? I use RTX 2080Ti to train the model.

Thanks a lot.

[chrischoy]

The MinkowskiEngine was significantly updated recently which speeds up the inference quite a bit. The update caches some data on the GPU which also improves the GPU utilization.

Could you try the latest version of the MinkowskiEngine?

[tangbohu]

Hi @chrischoy Thanks for your sharing. I tried your code on 3DMatch dataset using the default configuration and found the training process is very slow. Specifically it took about one and a half hour for one epoch. (as you mentioned in the paper, you trained FCGF for 100 epochs, which means more than one week in my configuration). The GPU memory it took is only less than 5000 MB and GPU utility is less than 10% but CPU utility is high. I wonder is it normal situation and what's the most time-consuming part ? I use RTX 2080Ti to train the model.

Thanks a lot.

Hi @XuyangBai, Have you solved the efficiency issue? I adopted MinkowskiEngine v0.4.2, but the training is still quite slow.

[XuyangBai]

@tangbohu Sorry I haven't solved it, the training is still slow.

[gitouni]

I have updated the ME to v0.5 but met merge_sort CUDA error issue#67.