Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix rank_cut setting and minor problems #2

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

April-Cai
Copy link

@April-Cai April-Cai commented Nov 26, 2019

Hi Qingyao,

I fixed some problems, these are details:

  1. add missing num_layers setting for the bug appeared in DLA/main.py
    print("Created %d layers of %d units." % (model.hparams.num_layers, model.embed_size))

  2. add missing flag: self_test

  3. import six for the use of xrange in python3

  4. the most important problem:
    I found the rank_cut setting does not work when evaluate, because the generated test.ranklist file does not cover all documents under the condition that the rank_cut is equal to 10 all the time.

I modified the code as I understand, I trained 2k epochs, then eval in the original code, here is the result by trec_eval:
ndcg_cut_1 all 0.6971
ndcg_cut_3 all 0.6950
ndcg_cut_5 all 0.7076
ndcg_cut_10 all 0.7373
map all 0.5383
runid all RankLSTM
num_q all 6983
num_ret all 63481
num_rel all 123035
num_rel_ret all 50891

Then I used the modified code to generate new test.ranklist with the same model, here is the updated result by trec_eval:
ndcg_cut_1 all 0.7126
ndcg_cut_3 all 0.7101
ndcg_cut_5 all 0.7243
ndcg_cut_10 all 0.7639
map all 0.8443
runid all RankLSTM
num_q all 6983
num_ret all 165660
num_rel all 123035
num_rel_ret all 123035

You can clearly see the difference in the two test.ranklist files.(the doc numbers are different)
The performance might be higher than results in the paper due to the use of a different dataset. But I am wondering is it the code version that your result reached in the original paper. If that, then it can be underestimated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant