论文复现问题

[huwenlong-cmd]

为什么复现这篇论文的程序时，评估模式之后的预测结果和训练模型当中计算的评价结果相差甚远？

[yuzhimanhua]

Thank you for your interest in our work! Could you provide more details? For example: (1) Which dataset are you using? (2) What are the numbers reported by our code during training? (3) What are the numbers calculated from the prediction results, and how do you calculate them?

[huwenlong-cmd]

使用的数据集（即您论文当中所使用的）：MAG 按照您提供的源程序，修改一些基本语法后进行训练，在训练过程中，报告的数值一切正常。问题出现在经过评估模式以及最后的evaluation.py之后，结果相差太大。或许是我水平有限，还没有找到问题所在。都是按照您提供的源程序中的方式计算数值。

…

------------------ 原始邮件 ------------------ 发件人: "yuzhimanhua/MATCH" ***@***.***>; 发送时间: 2024年7月14日(星期天) 下午5:23 ***@***.***>; ***@***.******@***.***>; 主题: Re: [yuzhimanhua/MATCH] 论文复现问题 (Issue #8) Thank you for your interest in our work! Could you provide more details? For example: (1) Which dataset are you using? (2) What are the numbers reported by our code during training? (3) What are the numbers calculated from the prediction results, and how do you calculate them? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

[yuzhimanhua]

I just ran the code on my side and did not observe this. Could you show me your command line output? Thanks!

[huwenlong-cmd]

感谢老师，我直接把源程序发您，请您帮我看看，问题就出现在评估阶段以及最后的预测。没想到您会直接回复我，真的太开心了。

…

------------------ 原始邮件 ------------------ 发件人: "yuzhimanhua/MATCH" ***@***.***>; 发送时间: 2024年7月15日(星期一) 凌晨3:27 ***@***.***>; ***@***.******@***.***>; 主题: Re: [yuzhimanhua/MATCH] 论文复现问题 (Issue #8) I just ran the code on my side and did not observe this. Could you show me your command line output? Thanks! — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***> 从QQ邮箱发来的超大附件 MATCH (2).zip (52.5K, 无限期)进入下载页面：https://mail.qq.com/cgi-bin/ftnExs_download?k=7a3733371d4bc89e1038e1451665054e4043520656070f021505030e5548005200011e0e045d0f4c08545500010652530c0e06013068372c7963707f104d0548164d5a473058&t=exs_ftn_download&code=87370e7a

[yuzhimanhua]

It would be much better if you ran the code on your side and showed me the command line output, so that I could know what happened on your side. That being said, I tried to run your code. Since you did not include a .sh script, I used my original one and removed your hard-coded parameters (i.e., Lines 19-22 in main.py and Lines 17-23 in preprocess.py). Still, I did not observe any problems.

Please note that:
(1) The reported P@k and NDCG@k scores during training are on the validation set; the final calculated P@k and NDCG@k scores during evaluation are on the testing set. Please expect a small gap (0.5%-1%) between these numbers. For example, on my side, the validation P@1 is 0.9211; the testing P@1 is 0.9150.

(2) I do not know how you wrote your .sh script. If your parameters are hard-coded, please remember to change them before you test the model.

If you have further questions, please show me your command line output. A fragment of mine is shown below:

Thanks!