Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

error finding in training #1

Open
Aoshika123 opened this issue Feb 24, 2023 · 3 comments
Open

error finding in training #1

Aoshika123 opened this issue Feb 24, 2023 · 3 comments

Comments

@Aoshika123
Copy link

Hello, thank you for your work. I modified the batchsize to 12 before training, and then error finding occurred after a period of time. Did the author encounter this problem before? Is it because of the lr setting problem?

@zhao1f
Copy link
Member

zhao1f commented Feb 24, 2023

Hi, I have tested it on several different machines and found it works well. Besides the commonly occurring Nan training in other codes, one possible problem here might be the abnormal values in the object part discovery. Slightly modifying the learning rate or batch size may help this issue.

If these do not help, modifying the hyper-parameters of part discovery (part number and queue number) may help.

@Carinazhao22
Copy link

Did you solve the problem? I also met the same problem.

@zhao1f
Copy link
Member

zhao1f commented Sep 30, 2023

Hi, modifying the hyper-parameters of part discovery (part number and queue number) may help.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants