9753 nan nan 0.00 nan nan 1698.001 16280.8

[mtli77]

Hi @XuyangBai

I'm sorry to disturb you with the following questions. In the process of 3dmatch training, in 9753 step

Steps desc_loss det_loss train_accuracy d_pos d_neg time memory
9753 nan nan 0.00 nan nan 1698.001 16280.8.

What does this mean when the desc_loss and det_loss is nan, and the train_accuracy become 0.00, is it some kind of error in the training process?

Thank you very much and waiting for your reply!

[XuyangBai]

Hi @Violetit It seems very similar to the problem in here. You may check your TF and CUDA version first.

[mtli77]

Hi @Violetit It seems very similar to the problem in here. You may check your TF and CUDA version first.

My CUDA version is 10.0 with the tensorflow-gpu=1.13.2