[Doc] Add documentation for Structured Outputs

[ismael-dm]

Adding some documentation on the structured outputs (guided decoding) options that are available, together with two files of examples. Includes both online inference (OpenAI API) and offline inference.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀

[simon-mo]

Nice! Thank you for the PR.

• Please add a link to the online inference docs from the OpenAI server page
• Please make sure all the models are instruction tuned variant.

[ismael-dm]

@simon-mo sure, I added the link to OpenAI server page and changed the models to instruct variant (an re-tested the examples). But now a couple of automatic tests are failling, any idea why?

[ismael-dm]

@simon-mo all the tests are good now and the requested changes were applied

[tcapelle]

Hey, just found your PR as I am struggling with struct output on openai VLLM, do you have any insights on this?

But for some reason, when using a pydantic model I get a bad output and whithout it it works. My code is this:

import openai

MODEL_NAME = "internlm/internlm2_5-7b-chat"

vllm_client = openai.AsyncOpenAI(
    base_url="http://localhost:8000/v1",
    api_key="dummy")

class PersonInfo(BaseModel):
    name: str = Field(description="The name of the person")
    age: int = Field(description="The age of the person")

json_schema = PersonInfo.model_json_schema()

async def instruct_call(prompt: str = "My friend Pedro is 25 years old", **kwargs):
    response = await vllm_client.chat.completions.create(
        model=MODEL_NAME,
        messages=[
            {"role": "system", "content": "You are a helpful assistant that generates JSON objects."},
            {"role": "user", "content": "Get the name and age from the following text: " + prompt}],
        **kwargs
    )
    return response.choices[0].message.content

print("="*80)
print("Structured output asking for JSON")
print(asyncio.run(instruct_call()))
print("="*80)
print("Structured output with JSON schema")
print(asyncio.run(instruct_call(extra_body={"guided_json": json_schema})))

# ================================================================================
# Structured output asking for JSON
#
# {
#   "name": "Pedro",
#   "age": 25
# }
#
# ================================================================================
# Structured output with JSON schema
# {"name":"Weird","age":25}```

[ismael-dm]

Hey, just found your PR as I am struggling with struct output on openai VLLM, do you have any insights on this?

But for some reason, when using a pydantic model I get a bad output and whithout it it works. My code is this:

import openai

MODEL_NAME = "internlm/internlm2_5-7b-chat"

vllm_client = openai.AsyncOpenAI(
    base_url="http://localhost:8000/v1",
    api_key="dummy")

class PersonInfo(BaseModel):
    name: str = Field(description="The name of the person")
    age: int = Field(description="The age of the person")

json_schema = PersonInfo.model_json_schema()

async def instruct_call(prompt: str = "My friend Pedro is 25 years old", **kwargs):
    response = await vllm_client.chat.completions.create(
        model=MODEL_NAME,
        messages=[
            {"role": "system", "content": "You are a helpful assistant that generates JSON objects."},
            {"role": "user", "content": "Get the name and age from the following text: " + prompt}],
        **kwargs
    )
    return response.choices[0].message.content

print("="*80)
print("Structured output asking for JSON")
print(asyncio.run(instruct_call()))
print("="*80)
print("Structured output with JSON schema")
print(asyncio.run(instruct_call(extra_body={"guided_json": json_schema})))

# ================================================================================
# Structured output asking for JSON
#
# {
#   "name": "Pedro",
#   "age": 25
# }
#
# ================================================================================
# Structured output with JSON schema
# {"name":"Weird","age":25}```

Not sure tbh. It seems that it´s generating a valid json, but the values are not properly filled. What I´ve seen is that some models perform better when filling out the fields of structured outputs. I´ve had the best results with Qwen models. Also, trying to iterate a little bit over the prompt may help, like giving some examples of the desired output. Are you using the latest vllm version? If any of these things work maybe the best option is to open an issue to discuss further.

[tcapelle]

I feel we are missing the openai.beta api example that gives you real structured output. https://x.com/dottxtai/status/1858495878463389877