[FEA]: Byte-Pair Encoder (BPE) support

[pdmack]

Is this a new feature, an improvement, or a change to existing functionality?

New Feature

How would you describe the priority of this feature request

Medium

Please provide a clear description of problem this feature solves

There are projects such as Bertin whose focus is to train and evaluate BERT-based models for the Spanish language. Models such as roberta, gpt-2, and gpt-3 require a BPE tokenizer. However, the current Morpheus inference pipeline currently uses the cuDF BERT tokenizer.

Describe your ideal solution

Develop a new CPU tokenizer/nlp-preprocessing stage for Morpheus. An adaptation of Morpheus current phishing training.

Describe any alternatives you have considered

cuDF project has an open feature request for GPU-accelerated BPE tokenizer support but without any known roadmap commitment.

Additional context

• Useful for phishing detection in non-English content
• https://leimao.github.io/blog/Byte-Pair-Encoding/
• https://towardsdatascience.com/byte-pair-encoding-subword-based-tokenization-algorithm-77828a70bee0

Code of Conduct

• I agree to follow this project's Code of Conduct
• I have searched the open feature requests and have found no duplicates for this feature request