Add support for GPT-4o tokenizer

[JonathanVelkeneers]

Support for OpenAI's GPT-4o should be added.
Upstream tokenizer dotnet/machinelearning#7154 has implemented support for o200k_base.

Changes:

• Bump Microsoft.ML.Tokenizers to at least 0.22.0-preview.24271.1
• Create new ITextTokenizer TikTokenGPT4oTokenizer which uses the new TikToken tokenizer

I'm willing to PR these changes

[jkone27]

Does also hugging face need a specific tokenizer? i get this error while trying semantic kernel hugging face, which then later fails

Microsoft.KernelMemory.SemanticKernel.SemanticKernelTextEmbeddingGenerator[0]
      Tokenizer not specified, will use GPT4Tokenizer. 

[dluc]

that's only a warning. You should always choose a tokenizer, but if you don't the system will use GPT4 tokenizer by default. The tokenizer is used to check the size of chunks when parsing documents and calculating embeddings. If the tokenizer chosen provided incorrect numbers, you might get runtime errors. When using HuggingFace, the tokenizer still depends on which model you're choosing. If you don't know what to do, keep the default and watch for errors that might be related to chunk size or embedding limitations.

[dluc]

here's the tokenizer https://github.com/microsoft/kernel-memory/blob/main/extensions/OpenAI/OpenAI/Tokenizers/GPT4oTokenizer.cs