TokenTextEncoder does not respect reserved tokens

[rsennrich]

I get poor results for translate_ende_wmt_bpe32k (this may be related to issues #317 , #309 ), and believe this is due to a problem with reserved tokens not being respected by TokenTextEncoder, and the provided vocabulary file.

INFO:tensorflow:Inference results INPUT: In this sense , the measures will partially undermine the American democratic system .
INFO:tensorflow:Inference results OUTPUT: In In In In In In In In In In In In In In In In In In In In In In In In In In In In In In In In In In In In In In In In In In In In In In In In In In In In In In In In In In In In In In In In In In In In In In In In In In In In In In In In
In In In In In

{'outputs': array([68, 68, 68, 68, 68, 68, 68, 68, 68, 68, 68, 68, 68, 68, 68, 68, 68,
68, 68, 68, 68, 68, 68, 68, 68, 68, 68, 68, 68, 68, 68, 68, 68, 68,
68, 68, 68, 68, 68, 68, 68, 68, 68, 68, 68, 68, 68, 68, 68, 68, 68,
68, 68, 68, 68, 68, 68, 68, 68, 68, 68, 68, 68, 68, 68, 68, 68, 68,
68, 68, 68, 68, 68, 68, 68, 68, 68, 68, 68, 68, 68, 68, 68, 68, 68], dtype=int32), 'problem_choice': 0, 'inputs': array([[ 68],
[ 35],
[ 2196],
[ 0],
[ 2],
[ 651],
[ 55],
[18587],
[15840],
[ 2],
[ 1874],
[ 1763],
[ 260],
[ 1],
[ 1],
[ 0],
[ 0],
[ 0],
[ 0],
[ 0],
[ 0],
[ 0],
[ 0],
[ 0],
[ 0],
[ 0],
[ 0],
[ 0],
[ 0],
[ 0],
[ 0],
[ 0],
[ 0],
[ 0],
[ 0]], dtype=int32)}

The first 10 lines of vocab.bpe.32000 look like this:

,
.
the
in
of
and
die
der
to
und

Note that "," and "." get assigned the indices 0 and 1, conflicting with the reserved tokens.

I started a new training run with a modified vocab:

pad
eos
,
.
the
in
of
and
die
der

and this gives better results (model hasn't converged yet):

INFO:tensorflow:Inference results INPUT: In this sense , the measures will partially undermine the American democratic system .
INFO:tensorflow:Inference results OUTPUT: In diesem Sinne wird die Maßnahmen in der Lage sein , das amerikanische System zu bekämpfen .

{'outputs': array([ 54, 11958, 11, 10408, 164, 231, 940, 92, 2802,
9, 12051, 7944, 3, 1, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0], dtype=int32), 'problem_choice': 0, 'inputs': array([[24820],
[ 22],
[ 3467],
[ 8753],
[ 111],
[ 322],
[ 48],
[ 4],
[ 229],
[ 10],
[ 4995],
[14095],
[ 7298],
[ 3],
[ 1],
[ 0],
[ 0],
[ 0],
[ 0],
[ 0],
[ 0],
[ 0],
[ 0],
[ 0],
[ 0],
[ 0],
[ 0],
[ 0],
[ 0],
[ 0],
[ 0],
[ 0],
[ 0],
[ 0],
[ 0]], dtype=int32)}

I'd suggest that TokenTextEncoder actually reserves the first two integers for reserved tokens, and starts counting from 2, but I realize this will break compatibility with trained models. Alternatively, you could fix the vocabularies of pre-defined problems - I don't know how many are affected.

[vince62s]

are you using your regular BPE ?
with the out-of-the-box subwords script the vocab.endefr.32768 file is as follow:
EDIT: I added spaces for the first 2 because github was not displaying properly.

' < pad > '
' < EOS > '
', '
'.'
'the_'
'de_'
''
'''
'in_'
'of_'

[mehmedes]

They seem to refer to regular BPE.
My BPE vocab also starts with , and . like Rico's but I didn't experience any issues with not including pad and eos. I trained my BPE model back on T2T 1.0.11.

[rsennrich]

I'm using the problem translate_ende_wmt_bpe32k , which out-of-the-box downloads and uses vocab.bpe.32000 (in v1.2.4).

[mehmedes]

@rico: Just wondering how did you manage to get a translation result in your second example using a BPE model. The input has only been tokenized and not preprocessed with BPE. When using the BPE model I think T2T requires you to tokenize as well as apply BPE before submitting for translation, and postprocess the inference result reverting BPE and detokenizing.

[lukaszkaiser]

Does this problem still exist in 1.2.5? We tried to correct the reserved tokens in vocabs, but maybe there's still sth missing?

[rsennrich]

@mehmedes : the preprocessed test sets are in /tmp/t2t_datagen (or wherever you downloaded wmt16_en_de.tar.gz ), for instance as newstest2013.tok.bpe.32000.en . From a quick look at the data, the files were (probably) preprocessed like this:

/path/to/mosesdecoder/scripts/tokenizer/normalize-punctuation.perl -l en | \
/path/to/mosesdecoder/scripts/tokenizer/tokenizer.perl -l en | \
/path/to/subword-nmt/apply_bpe.py -c bpe.32000

@lukaszkaiser : there's been no change to the vocabs or the handling of reserved tokens between 1.2.4 and 1.2.5 - yes, the problem still persists.

[mehmedes]

I was asking because your input stated a clean sentence

INFO:tensorflow:Inference results INPUT: In this sense , the measures will partially undermine the American democratic system .

I would have expected the input to be more like

INFO:tensorflow:Inference results INPUT: In th@@ sen@@ etc. 

[rsennrich]

Coincidentally, all words in this sentence happen to be frequent in the training data.

[rsepassi]

The behavior of TokenTextEncoder with regards to reserved tokens depends on whether it is constructed from a file or from a list. If it is constructed from a file, then it's expected that the file includes reserved tokens (i.e. the first line is <pad> and the second line is <EOS>) if there are any. If it's from a list, then the reserved tokens are added. The reason is because if you initialize from a list and then call store_to_file, the file will include the reserved tokens.

[lkluo]

I am using the latest version without reported problem, when I adopted TokenTextEncoder and external subword.
BTW, have anyone tested which is better for NMT task, the internal wordpiece or external subword?

[colmantse]

what mean wordpiece and subword, you mean bpe and t2t's default?

[lkluo]

@colmantse Yes, that's exactly what I meant. It seems wordpiece has better control over vocabulary size.

[colmantse]

bpe or t2t's default? I find t2t's default better on en-zh.

[lkluo]

Good to know. Have you processed Chinese sentence with segmentation?

…

On 17 Jan 2018 22:16, "colmantse" ***@***.***> wrote: bpe or t2t's default? I find t2t's default better on en-zh. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#365 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AY0L4C9C_KS9CuW1MIng-uQWcTx0LTxmks5tLgDHgaJpZM4P6x0C> .

[colmantse]

I didnt preprocess them. Judging by performance, it works just fine. 取得 Outlook for Android On Wed, Jan 17, 2018 at 10:39 PM +0800, "lkluo" <[email protected]> wrote: Good to know. Have you processed Chinese sentence with segmentation?

On 17 Jan 2018 22:16, "colmantse" ***@***.***> wrote: bpe or t2t's default? I find t2t's default better on en-zh. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#365 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AY0L4C9C_KS9CuW1MIng-uQWcTx0LTxmks5tLgDHgaJpZM4P6x0C> .

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.