Cannot reproduce "transformer" paper's result with BPE32k

[skyw]

I can train translate_ende_wmt32k and get reasonable bleu with transformer_big model. But if I train translate_ende_wmt_bpe32k, it doesn't translate anything meaningful.

I've tried several times, all with default settings. Any suggestions?

[mehmedes]

Have you made sure that you preprocessed the translatable text in the same way like the training data? You'll need to tokenize and apply bpe before translating.

[skyw]

I believe I tried both newstest2015.en, and newstest2015.tok.clean.bpe.32000.en.
let me confirm and report back.

[skyw]

Here is what I got. Most paper reports bleu scores 0~100, is this really 0.31 or 31.3?
translate_ende_wmt_bpe32k/approx_bleu_score = 0.313501, metrics-translate_ende_wmt_bpe32k/rouge_2_fscore = 0.398882, metrics-translate_ende_wmt_bpe32k/rouge_L_fscore = 0.593164

[martinpopel]

The original BLEU was defined as 0-1, but most people report it as 0-100. Note that approx_bleu is not BLEU (usually it is about 1.5 times higher).

[skyw]

0.313501 / 1.5 is still a reasonable number, I suppose