Subword Tokenizer HuggingFace like API

[VibhuJawa]

This PR closes #5868 by adding a new tokenizer API.

We are seeing speedups even at low batch sizes (10/100) so this should potentially unlock some inference/training use cases for us.

Benchmarks:

(Thanks to @davidwendt for writing the super fast tokenizer 💥 )

Batch Size	HuggingFace	Rapids Old API	Rapids Tokenizer API	Tokenizer API Speed up vs HuggingFace	Rapids New API Speedup
1	0.000242	0.006890	0.000497	0.487	13.863
10	0.002800	0.007030	0.000516	5.426	13.624
100	0.016200	0.007140	0.000537	30.168	13.296
1000	0.149000	0.007150	0.000517	288.201	13.830

API Comparision to HuggingFace:

The goal of this PR is to ensure our API matches up HuggingFace as much as possible to help with ease of porting.

Proposed API in this PR:

from cudf.core.subword_tokenizer import SubwordTokenizer
tokenizer = SubwordTokenizer('bert-base-cased-vocab-hash.txt',do_lower_case=False)

output = tokenizer(str_series,
                   max_num_rows=len(str_series),
                   truncation=True,
                   max_length=seq_len,
                   padding='max_length',
                   add_special_tokens=False,
                   return_tensors='pt')

HuggingFace API:

from transformers import BertTokenizerFast
tokenizer = BertTokenizerFast.from_pretrained('bert-base-cased', do_lower_case=False)

output = tokenizer(input_sentence_ls,
                   truncation=True,
                   max_length=seq_len,
                   padding='max_length',
                   add_special_tokens=False,
                   return_tensors = 'pt')
output_d = {k:v.cuda() for k,v in output.items()} 

TODO:

• Add tests
• Throw appropriate warnings for HuggingFace discrepancies
• API checks
• Benchmark/Example Notebook

CC: @raykallen, @BartleyR (from the cyber team)

CC: @randerzander , @beckernick (from the workflows team)

[VibhuJawa]

Changed DataFiles to allow for testing against HuggingFace.

[VibhuJawa]

Also, is it necessary to change the text files used for testing here? Changing big files like this just blows up the git repository size over time.

We previously had uncased vocabulary which was randomly sampled making it difficult to run tests with it. This new vocabulary is a cased one and does meaningful tokenization. We can probably just download them on the fly if making this is prohibitive. FWIW, I don't expect this file to change again in the future.

[ajschmidt8]

approved now that rapidsai/integration#249 is merged.

[VibhuJawa]

@kkraus14, @galipremsagar. Thanks a ton for your detailed reviews.

I think all reviews have been addressed, and this PR is ready for another review.

[kkraus14]

@gpucibot merge

[galipremsagar]

rerun tests

[VibhuJawa]

rerun tests