Addition of Tokenization API

[GirinMan]

Feature request

• I propose the addition of a /tokenize route to the API, allowing users to obtain tokenization results directly through the API.
• This feature is inspired by the functionality seen in HF TGI, which introduced a similar feature.
• Implementing this in LoRAX would greatly enhance its utility and align it with current advancements in NLP APIs.

Motivation

• The motivation behind this feature request is to streamline the process for users to get tokenization results without the need for external tools.
• Tokenization is a fundamental step in many NLP tasks, and integrating this functionality directly into our API would significantly improve user experience.
• This feature request is not directly related to a problem but is an enhancement to expand the API's capabilities.

Your contribution

• I am prepared to contribute to this feature by submitting a PR that introduces the /tokenize route to the API, along with necessary documentation and tests to ensure its functionality and reliability.