Regex decoding support

[sheepymeh]

Is regex decoding or a "JSON mode" available?

[yufenglee]

Could you please explain more on this? What is regex decoding?

[sheepymeh]

It's decoding that follows a specific format (typically described by a regex). Some examples are:

Outlines: https://github.com/outlines-dev/outlines

SGLang: https://lmsys.org/blog/2024-02-05-compressed-fsm/

This allows the model to generate according to a fixed format that can later be decoded, such as ensuring that output is valid JSON.

[natke]

Thank you for the suggestion! I am adding it to our new Features discussions so that we can look into this further