Loss goes nan when training dual swin-base

[seanzhuh]

Hi, I've transferred your code to my own codebase, since your modification of the original mmdetection lies in 3 files (please correct me if I'm wrong):

1. CBNetV2/mmdet/models/backbones/cbnet.py
2. CBNetV2/mmdet/models/necks/cbnet_fpn.py
3. CBNetV2/mmdet/models/detectors/two_stage.py

I directly do a copy-paste to transfer your code my own version of mmdetection, however, loss goes to nan since epoch 17, the reason is that gradient is overflowing, amp loss scaler has to shrink to a small number, until divided by zero, thus it goes to nan.

I use default setting of AMP as yours, I can't figure out what's wrong, could you help me?

[fuweifu-vtoo]

I meet the same situation!
have you solved it?

[seanzhuh]

Nope, I've tried to transfer CBNetV2 to MMdetection v2.12.0 but still goes NaN exactly at 17th epoch.

…

On Thu, 16 Sept 2021 at 11:40, fuweifu-vtoo ***@***.***> wrote: I meet the same situation! have you solved it? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#39 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AG6FBLOZSE7ALADGAOWIMI3UCFRKDANCNFSM5CYBQLHA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.