Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Chore/update structured output documentation #1749

Merged
merged 5 commits into from
Dec 3, 2024
Merged

Chore/update structured output documentation #1749

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
65547d5
Merge branch 'dev' of github.com:janhq/cortex.cpp into dev
nguyenhoangthuan99 Nov 11, 2024
81592d8
Merge branch 'dev' of github.com:janhq/cortex.cpp into dev
nguyenhoangthuan99 Nov 19, 2024
4c7d4f3
Merge branch 'dev' of github.com:janhq/cortex.cpp into dev
nguyenhoangthuan99 Nov 21, 2024
79ecc37
Merge branch 'dev' of github.com:janhq/cortex.cpp into dev
nguyenhoangthuan99 Nov 29, 2024
208c3f5
chore: update structured output documentation base on new cortex.llam…
nguyenhoangthuan99 Nov 29, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
264 changes: 102 additions & 162 deletions docs/docs/guides/structured-outputs.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,17 +5,68 @@ title: Structured Outputs

Structured outputs, or response formats, are a feature designed to generate responses in a defined JSON schema, enabling more predictable and machine-readable outputs. This is essential for applications where data consistency and format adherence are crucial, such as automated data processing, structured data generation, and integrations with other systems.

In recent developments, systems like OpenAI's models have excelled at producing these structured outputs. However, while open-source models like Llama 3.1 and Mistral Nemo offer powerful capabilities, they currently struggle to produce reliably structured JSON outputs required for advanced use cases. This often stems from the models not being specifically trained on tasks demanding strict schema adherence.
In recent developments, systems like OpenAI's models have excelled at producing these structured outputs. However, while open-source models like Llama 3.1 and Mistral Nemo offer powerful capabilities, they currently struggle to produce reliably structured JSON outputs required for advanced use cases.

This guide explores the concept of structured outputs using these models, highlights the challenges faced in achieving consistent output formatting, and provides strategies for improving output accuracy, particularly when using models that don't inherently support this feature as robustly as GPT models.

By understanding these nuances, users can make informed decisions when choosing models for tasks requiring structured outputs, ensuring that the tools they select align with their project's formatting requirements and expected accuracy.

The Structured Outputs/Response Format feature in [OpenAI](https://platform.openai.com/docs/guides/structured-outputs) is fundamentally a prompt engineering challenge. While its goal is to use system prompts to generate JSON output matching a specific schema, popular open-source models like Llama 3.1 and Mistral Nemo struggle to consistently generate exact JSON output that matches the requirements. An easy way to directly guild the model to reponse in json format in system message:
The Structured Outputs/Response Format feature in [OpenAI](https://platform.openai.com/docs/guides/structured-outputs) is fundamentally a prompt engineering challenge. While its goal is to use system prompts to generate JSON output matching a specific schema, popular open-source models like Llama 3.1 and Mistral Nemo struggle to consistently generate exact JSON output that matches the requirements. An easy way to directly guild the model to reponse in json format in system message, you just need to pass the pydantic model to `response_format`:

```
from pydantic import BaseModel
from openai import OpenAI
import json
ENDPOINT = "http://localhost:39281/v1"
MODEL = "llama3.1:8b-gguf-q4-km"

client = OpenAI(
base_url=ENDPOINT,
api_key="not-needed"
)


class CalendarEvent(BaseModel):
name: str
date: str
participants: list[str]


completion = client.beta.chat.completions.parse(
model=MODEL,
messages=[
{"role": "system", "content": "Extract the event information."},
{"role": "user", "content": "Alice and Bob are going to a science fair on Friday."},
],
response_format=CalendarEvent,
stop=["<|eot_id|>"]
)

event = completion.choices[0].message.parsed

print(json.dumps(event.dict(), indent=4))
```

The output of the model like this

```
{
"name": "science fair",
"date": "Friday",
"participants": [
"Alice",
"Bob"
]
}
```

With more complex json format, llama3.1 still struggle to response correct answer:

```

from openai import OpenAI
from pydantic import BaseModel
import json
ENDPOINT = "http://localhost:39281/v1"
MODEL = "llama3.1:8b-gguf-q4-km"
client = OpenAI(
Expand All @@ -39,203 +90,92 @@ completion_payload = {
]
}

response = client.chat.completions.create(

class Step(BaseModel):
explanation: str
output: str


class MathReasoning(BaseModel):
steps: list[Step]
final_answer: str


response = client.beta.chat.completions.parse(
top_p=0.9,
temperature=0.6,
model=MODEL,
messages=completion_payload["messages"]
messages=completion_payload["messages"],
stop=["<|eot_id|>"],
response_format=MathReasoning
)

print(response)
math_reasoning = response.choices[0].message.parsed
print(json.dumps(math_reasoning.dict(), indent=4))
```

The output of the model like this
The output of model looks like this

```

ChatCompletion(
id='OZI0q8hghjYQY7NXlLId',
choices=[
Choice(
finish_reason=None,
index=0,
logprobs=None,
message=ChatCompletionMessage(
content='''Here's how you can solve it:

{
"steps": [
{
"explanation": "First, we need to isolate the variable x. To do this, subtract 7 from both sides of the equation.",
"explanation": "To isolate the variable x, we need to get rid of the constant term on the left-hand side. We can do this by subtracting 7 from both sides of the equation.",
"output": "8x + 7 - 7 = -23 - 7"
},
{
"explanation": "This simplifies to 8x = -30",
"explanation": "Simplifying the left-hand side, we get:",
"output": "8x = -30"
},
{
"explanation": "Next, divide both sides of the equation by 8 to solve for x.",
"output": "(8x) / 8 = -30 / 8"
"explanation": "Now, to solve for x, we need to isolate it by dividing both sides of the equation by 8.",
"output": "8x / 8 = -30 / 8"
},
{
"explanation": "This simplifies to x = -3.75",
"explanation": "Simplifying the right-hand side, we get:",
"output": "x = -3.75"
}
],
"final_output": "-3.75"
}''',
refusal=None,
role='assistant',
audio=None,
function_call=None,
tool_calls=None
)
)
],
created=1730645716,
model='_',
object='chat.completion',
service_tier=None,
system_fingerprint='_',
usage=CompletionUsage(
completion_tokens=190,
prompt_tokens=78,
total_tokens=268,
completion_tokens_details=None,
prompt_tokens_details=None
)
)
"final_answer": "There is no final answer yet, let's break it down step by step."
}
```

From the output, you can easily parse the response to get correct json format as you guild the model in the system prompt.
Even if the model can generate correct format but the information doesn't 100% accurate, the `final_answer` should be `-3.75` instead of `There is no final answer yet, let's break it down step by step.`.

Howerver, open source model like llama3.1 or mistral nemo still truggling on mimic newest OpenAI API on response format. For example, consider this request created using the OpenAI library with very simple request like [OpenAI](https://platform.openai.com/docs/guides/structured-outputs#chain-of-thought):
Another usecase for structured output with json response, you can provide the `response_format={"type" : "json_object"}`, the model will be force to generate json output.

```
from openai import OpenAI
ENDPOINT = "http://localhost:39281/v1"
MODEL = "llama3.1:8b-gguf-q4-km"
client = OpenAI(
base_url=ENDPOINT,
api_key="not-needed"
)

class Step(BaseModel):
explanation: str
output: str


class MathReasoning(BaseModel):
steps: List[Step]
final_answer: str


completion_payload = {
"messages": [
{"role": "system", "content": f"You are a helpful math tutor. Guide the user through the solution step by step.\n"},
{"role": "user", "content": "how can I solve 8x + 7 = -23"}
]
}

response = client.beta.chat.completions.parse(
top_p=0.9,
temperature=0.6,
json_format = {"song_name":"release date"}
completion = client.chat.completions.create(
model=MODEL,
messages= completion_payload["messages"],
response_format=MathReasoning
messages=[
{"role": "system", "content": f"You are a helpful assistant, you must reponse with this format: '{json_format}'"},
{"role": "user", "content": "List 10 songs for me"}
],
response_format={"type": "json_object"},
stop=["<|eot_id|>"]
)
```

The response format parsed by OpenAI before sending to the server is quite complex for the `MathReasoning` schema. Unlike GPT models, Llama 3.1 and Mistral Nemo cannot reliably generate responses that can be parsed as shown in the [OpenAI tutorial](https://platform.openai.com/docs/guides/structured-outputs/example-response). This may be due to these models not being trained on similar structured output tasks.

```
"response_format" :
{
"json_schema" :
{
"name" : "MathReasoning",
"schema" :
{
"$defs" :
{
"Step" :
{
"additionalProperties" : false,
"properties" :
{
"explanation" :
{
"title" : "Explanation",
"type" : "string"
},
"output" :
{
"title" : "Output",
"type" : "string"
}
},
"required" :
[
"explanation",
"output"
],
"title" : "Step",
"type" : "object"
}
},
"additionalProperties" : false,
"properties" :
{
"final_answer" :
{
"title" : "Final Answer",
"type" : "string"
},
"steps" :
{
"items" :
{
"$ref" : "#/$defs/Step"
},
"title" : "Steps",
"type" : "array"
}
},
"required" :
[
"steps",
"final_answer"
],
"title" : "MathReasoning",
"type" : "object"
},
"strict" : true
},
"type" : "json_schema"
}
print(json.dumps(json.loads(completion.choices[0].message.content), indent=4))
```

The response for this request by `mistral-nemo` and `llama3.1` can not be used to parse result like in the [original tutorial by openAI](https://platform.openai.com/docs/guides/structured-outputs/example-response). Maybe `llama3.1` and `mistral-nemo` didn't train with this kind of data, so it fails to handle this case.
The output will looks like this:

```
Response: {
"choices" :
[
{
"finish_reason" : null,
"index" : 0,
"message" :
{
"content" : "Here's a step-by-step guide to solving the equation 8x + 7 = -23:\n\n```json\n{\n \"name\": \"MathReasoning\",\n \"schema\": {\n \"$defs\": {\n \"Step\": {\n \"additionalProperties\": false,\n \"properties\": {\n \"explanation\": {\"title\": \"Explanation\", \"type\": \"string\"},\n \"output\": {\"title\": \"Output\", \"type\": \"string\"}\n },\n \"required\": [\"explanation\", \"output\"],\n \"title\": \"Step\",\n \"type\": \"object\"\n }\n },\n \"additionalProperties\": false,\n \"properties\": {\n \"final_answer\": {\"title\": \"Final Answer\", \"type\": \"string\"},\n \"steps\": {\n \"items\": {\"$ref\": \"#/$defs/Step\"},\n \"title\": \"Steps\",\n \"type\": \"array\"\n }\n },\n \"required\": [\"steps\", \"final_answer\"],\n \"title\": \"MathReasoning\",\n \"type\": \"object\"\n },\n \"strict\": true\n}\n```\n\n1. **Subtract 7 from both sides** to isolate the term with x:\n\n - Explanation: To get rid of the +7 on the left side, we add -7 to both sides of the equation.\n - Output: `8x + 7 - 7 = -23 - 7`\n\n This simplifies to:\n ```\n 8x = -30\n ```\n\n2. **Divide both sides by 8** to solve for x:\n\n - Explanation: To get rid of the 8 on the left side, we multiply both sides of the equation by the reciprocal of 8, which is 1/8.\n - Output: `8x / 8 = -30 / 8`\n\n This simplifies to:\n ```\n x = -3.75\n ```\n\nSo, the final answer is:\n\n- Final Answer: `x = -3.75`",
"role" : "assistant"
}
}
],
{
"Happy": "2013",
"Uptown Funk": "2014",
"Shut Up and Dance": "2014",
"Can't Stop the Feeling!": "2016",
"We Found Love": "2011",
"All About That Bass": "2014",
"Radioactive": "2012",
"SexyBack": "2006",
"Crazy": "2007",
"Viva la Vida": "2008"
}
```




## Limitations of Open-Source Models for Structured Outputs

While the concept of structured outputs is compelling, particularly for applications requiring machine-readable data, it's important to understand that not all models support this capability equally. Open-source models such as Llama 3.1 and Mistral Nemo face notable challenges in generating outputs that adhere strictly to defined JSON schemas. Here are the key limitations:
Expand Down
Loading