creating custom tokenizer models in python

[ctruexcytiva]

Hello,

I would like to create a custom tokenizer model using the library drain3 or use similar methods. Looking at the examples found in bindings/python/examples/custom_components.py , I see that it isn't hard to create a pre-tokenizer or normalizer, but couldn't follow the same pattern of using the custom method. would I just need to implement a class such as the following and initialize the tokenizer class with Tokenizer(MyCustomModelClass())?

class Drain3TokenizationModel:
    def __init__(self):
        pass

    def __getstate__(self, /):
        pass

    def __setstate__(self, /, state):
        pass

    def get_trainer(self):
        return Trainer()

    def id_to_token(self, id):
        return "token"
    
    def save(self, folder, prefix):
        return []

    def token_to_id(self, tokens):
        return 1

    def tokenize(self, sequence):
        return [tokenizers.Token(1, "token", (i, i+1)) for i, _ in enumerate(sequence)]

    @staticmethod
    def __new__(*args, **kwargs):
        return Drain3TokenizationModel()

[ctruexcytiva]

Just to add to this, are the tokenizer models only implemented in rust and to add to this would require a rust implementation? Would it be better to try and make a subclass of the TokenizerBase class in python?