General Distill Stage?

[lsyysl9711]

Hello! The work is great and thanks for sharing the codes!

But I am confused about the general distill stage: In the Bert-emd folder, I see a file called "general_distill" but it seems that the file is not used and also can you tell me what is the pregenerate-dataset in general_distill.py?

Than you very much!

[lxk00]

This work draws on TinyBERT and we reuse some of the code. However, we did not perform general distillation and directly use TinyBERT released model. For General Distill Stage, you can refer to https://github.com/huawei-noah/Pretrained-Language-Model/tree/master/TinyBERT.