[kushalkodnad/tokenizer-registry] Introduce new registry for tokenizers

[kushalkodn-db]

What Does This PR Do?

The purpose of this PR is to introduce a new registry, specifically for tokenizers. This way, in the llmfoudry/utils/builders.py build_tokenizer() method, we can build a tokenizer from the registry, as long as the tokenizer inherits the transformers.PreTrainedTokenizerBase interface. For instance, the existing TiktokenTokenizerWrapper in llmfoundry/tokenizers/tiktoken.py can be built from this registry, as opposed to using if-else clauses in build_tokenizer, which can become tedious if we add support for more tokenizers.

File-Specific Changes

llmfoundry/registry.py

This is where I created the tokenizers registry. See the brief description of the llmfoundry tokenizers registry, defined in _tokenizers_description. Then, I followed the callbacks registry creation to do the same thing for tokenizers when calling create_registry.

llmfoundry/utils/builders.py

The main idea is to replace the if-else clause that currently supports "tiktoken". As long as the tokenizer is a sub-class of transformers.PreTrainedTokenizerBase, then the tokenizer will be constructed.

llmfoundry/tokenizers/__init__.py

Following the callbacks example, the addition to this file shows how to register a transformers.PreTrainedTokenizerBase sub-class to the tokenizers registry. Here, I registered the tiktoken tokenizer to the tokenizers registry, because it inherits the same interface.