You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hi,
when train the student network using soft logits method and running the code:
python3 train_w_distill2.py --Distillation=Soft_logits --train_dir=soft_logits --main_scope=Student_w_Soft_logits --teacher=ResNet32.mat ,I find that the results changes a lot . After training five times, I get 71.48, 72.25, 72.34, 71.73, 72.46 (Best Accuracy)for the student network. Is this normal?
The text was updated successfully, but these errors were encountered:
If you get an average of them, it is similar to my experimental results.
My results are
Last : [71.63, 71.80, 71.87, 71.66, 71.98]
Best : [71.89, 72.41, 71.97, 72.05, 72.10]
We rarely consider the best accuracy. I presented this just for additional information.
Focus on the last. In our case, they are not so fluctuated.
OK, Thank you for the reply. I got another question. We can see that the soft logits method doesn't get a pretty result( Best accuacy for student network from 71.76 to 71.79). Have you try other settings,?such as T=10 or higher coefficent for the KL loss
No, I didn't try so many times for each method. There are so many configurations for knowledge distillation, such as hyper-parameters, feature sensing points, and so on, which is hard to find the optimal one. If you want, you can find a better configuration for them.
Hi,
when train the student network using soft logits method and running the code:
python3 train_w_distill2.py --Distillation=Soft_logits --train_dir=soft_logits --main_scope=Student_w_Soft_logits --teacher=ResNet32.mat ,I find that the results changes a lot . After training five times, I get 71.48, 72.25, 72.34, 71.73, 72.46 (Best Accuracy)for the student network. Is this normal?
The text was updated successfully, but these errors were encountered: