Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The results of Soft Logits fluctuate quite a lot #18

Open
zhongshaoyy opened this issue Nov 19, 2019 · 3 comments
Open

The results of Soft Logits fluctuate quite a lot #18

zhongshaoyy opened this issue Nov 19, 2019 · 3 comments

Comments

@zhongshaoyy
Copy link

Hi,
when train the student network using soft logits method and running the code:
python3 train_w_distill2.py --Distillation=Soft_logits --train_dir=soft_logits --main_scope=Student_w_Soft_logits --teacher=ResNet32.mat ,I find that the results changes a lot . After training five times, I get 71.48, 72.25, 72.34, 71.73, 72.46 (Best Accuracy)for the student network. Is this normal?

@sseung0703
Copy link
Owner

If you get an average of them, it is similar to my experimental results.
My results are
Last : [71.63, 71.80, 71.87, 71.66, 71.98]
Best : [71.89, 72.41, 71.97, 72.05, 72.10]
We rarely consider the best accuracy. I presented this just for additional information.
Focus on the last. In our case, they are not so fluctuated.

@zhongshaoyy
Copy link
Author

OK, Thank you for the reply. I got another question. We can see that the soft logits method doesn't get a pretty result( Best accuacy for student network from 71.76 to 71.79). Have you try other settings,?such as T=10 or higher coefficent for the KL loss

@sseung0703
Copy link
Owner

No, I didn't try so many times for each method. There are so many configurations for knowledge distillation, such as hyper-parameters, feature sensing points, and so on, which is hard to find the optimal one. If you want, you can find a better configuration for them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants