Respect add_prefix_space option in LlamaTokenizerFast

[aoxolotl]

What does this PR do?

Respect add_prefix_space option in Llama tokenizer (Fixes #29625)

The add_prefix_space option in Llama tokenizer was set but not passed to the super().__init__() method to PretrainedTokenizersFast. This resulted in SPIECE_UNDERLINE token being added even when add_prefix_space=False.

Minimal example

When add_prefix_space is False

>>> from transformers import AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf", add_prefix_space=False)
>>> tokenizer.tokenize('overheard')
['over', 'he', 'ard']

add_prefix_space is True

>>> tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf", add_prefix_space=True)
>>> tokenizer.tokenize('overheard')
>>> ['\u2581over', 'he', 'ard']

Fixes #29625

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

@ArthurZucker @scruel

[ArthurZucker]

Thanks! Could you add a test to make sure this works? 🤗

[aoxolotl]

Sure! Will update the pull request with the tests

[aoxolotl]

Hey @ArthurZucker, I have a question about the tests. There already seems to be an add_prefix_space test present in test_tokenization_llama.py. However, this strangely passes even without the modifications above. The difference seems to come from using AutoTokenizer (as in the mentioned issue) vs using the LlamaTokenizer* classes directly. Is there a difference between the init routes we take in the above two scenarios?

Edit:
The difference actually comes from using hf-internal-testing/llama-tokenizer-non-normalized vs meta-llama/Llama-2-7b-hf. Examples:

>>> hf_tokenizer = LlamaTokenizerFast.from_pretrained("hf-internal-testing/llama-tokenizer-non-normalized", add_prefix_space=False, legacy=False)
>>> hf_tokenizer.tokenize('overheard')
['over', 'he', 'ard']

>>> tokenizer = LlamaTokenizerFast.from_pretrained("meta-llama/Llama-2-7b-hf", add_prefix_space=False, legacy=False)
>>> tokenizer.tokenize('overheard')
['\u2581over', 'he', 'ard']

[ArthurZucker]

Actually I think we need to wait for #28881, to have a proper fix for Llama! Feel free to skip it. T5 should work!

[scruel]

Won't you see my review?

[ArthurZucker]

pending means it's not been submitted!

[scruel]

pending means it's not been submitted!

Oh, I see, it this HF repos limited review action by others?

[ArthurZucker]

No, pressing the submit review button should be enough

[scruel]

Mind remove this empty line?

[aoxolotl]

Sure, updating in latest commit

[scruel]

No, pressing the submit review button should be enough

It works! Thanks!

[aoxolotl]

@scruel Removed in latest commit!
@ArthurZucker I will make a separate PR for T5 tokenizer which adds any necessary tests. Leaving this PR for LlamaTokenizer specific changes

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[ArthurZucker]

#28881 should actually fix this!