fssd divergent

[lucasjinreal]

Training FSSD_vgg on VOC0712
Epoch:1 || epochiter: 0/313|| Totel iter 0 || L: 0.3828 C: 1.9178||Batch time: 6.2867 sec. ||LR: 0.00400000
Epoch:1 || epochiter: 10/313|| Totel iter 10 || L: 3.0171 C: 10.5603||Batch time: 0.3471 sec. ||LR: 0.00400000
Epoch:1 || epochiter: 20/313|| Totel iter 20 || L: 2.8891 C: 7.1995||Batch time: 0.5090 sec. ||LR: 0.00400000
Epoch:1 || epochiter: 30/313|| Totel iter 30 || L: 2.8777 C: 8.0690||Batch time: 0.3880 sec. ||LR: 0.00400000
Epoch:1 || epochiter: 40/313|| Totel iter 40 || L: 2.9136 C: 7.9621||Batch time: 0.3389 sec. ||LR: 0.00400000
Epoch:1 || epochiter: 50/313|| Totel iter 50 || L: 2.8447 C: 7.2007||Batch time: 0.3387 sec. ||LR: 0.00400000
Epoch:1 || epochiter: 60/313|| Totel iter 60 || L: 2.9121 C: 8.6076||Batch time: 0.3385 sec. ||LR: 0.00400000
Epoch:1 || epochiter: 70/313|| Totel iter 70 || L: 2.9971 C: 9.7988||Batch time: 0.3462 sec. ||LR: 0.00400000
Epoch:1 || epochiter: 80/313|| Totel iter 80 || L: 3.6867 C: 14.9873||Batch time: 0.3368 sec. ||LR: 0.00400000
Epoch:1 || epochiter: 90/313|| Totel iter 90 || L: 7197.3708 C: 94863.1805||Batch time: 0.3424 sec. ||LR: 0.00400000
Epoch:1 || epochiter: 100/313|| Totel iter 100 || L: nan C: nan||Batch time: 0.3348 sec. ||LR: 0.00400000
Epoch:1 || epochiter: 110/313|| Totel iter 110 || L: nan C: nan||Batch time: 0.3376 sec. ||LR: 0.00400000
Epoch:1 || epochiter: 120/313|| Totel iter 120 || L: nan C: nan||Batch time: 0.3405 sec. ||LR: 0.00400000
Epoch:1 || epochiter: 130/313|| Totel iter 130 || L: nan C: nan||Batch time: 0.3446 sec. ||LR: 0.00400000
Epoch:1 || epochiter: 140/313|| Totel iter 140 || L: nan C: nan||Batch time: 0.3363 sec. ||LR: 0.00400000
Epoch:1 || epochiter: 150/313|| Totel iter 150 || L: nan C: nan||Batch time: 0.3784 sec. ||LR: 0.00400000
Epoch:1 || epochiter: 160/313|| Totel iter 160 || L: nan C: nan||Batch time: 0.3344 sec. ||LR: 0.00400000
Epoch:1 || epochiter: 170/313|| Totel iter 170 || L: nan C: nan||Batch time: 0.3403 sec. ||LR: 0.00400000
Epoch:1 || epochiter: 180/313|| Totel iter 180 || L: nan C: nan||Batch time: 0.3361 sec. ||LR: 0.00400000
Epoch:1 || epochiter: 190/313|| Totel iter 190 || L: nan C: nan||Batch time: 0.7398 sec. ||LR: 0.00400000
Epoch:1 || epochiter: 200/313|| Totel iter 200 || L: nan C: nan||Batch time: 0.3413 sec. ||LR: 0.00400000
Epoch:1 || epochiter: 210/313|| Totel iter 210 || L: nan C: nan||Batch time: 0.3358 sec. ||LR: 0.00400000
Epoch:1 || epochiter: 220/313|| Totel iter 220 || L: nan C: nan||Batch time: 0.3381 sec. ||LR: 0.00400000

[yosunpeng]

smaller learning rate works for me: --lr 1e-3