New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bert中的Embedding Layer #24

Open

loxs123 opened this issue Nov 18, 2022 · 1 comment

loxs123 commented Nov 18, 2022

您好，Bert当中的Embedding Layer是在Layer0之前的，他的学习率设置为Layer0乘以权重ξ（0.95）会不会更好一点？

Owner

xuyige commented Nov 18, 2022

您好，感谢您对我们repo的兴趣。
我们之前并没有研究Embedding Layer的learning rate decay问题，您可以尝试一下将learning rate decay加到Embedding Layer上面。
现在的learning rate设置在https://github.com/xuyige/BERT4doc-Classification/blob/master/codes/fine-tuning/run_classifier_discriminative.py#L853 这一段落，可以将带embedding的参数加进groups里面

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment