Issue with Chat Vision

[GitHubUsername1]

Might not be a bug, could just be me (probably is, I've tried to solve it myself, no luck).

Overview

I'm having an issue with Chat Vision, the response message is cut off. I went over the readme and seen this is due to the response generating unending whitespace because of not specifying in the system message the response should be Json, and also the enum formatting type should be Json, so I changed my code. However now I'm getting the error -

{
  "error": {
    "message": "Invalid content type. image_url is only supported by certain models.",
    "type": "invalid_request_error",
    "param": "messages.[1].content.[1].type",
    "code": null
    }
}

In the console.

To Reproduce

The code to create a response with unending whitespace in response -

var messages = new List<OpenAI.Chat.Message>
    {
        new OpenAI.Chat.Message(Role.System, "You are a helpful assistant."),
        new OpenAI.Chat.Message(Role.User, new List<OpenAI.Chat.Content>
        {
             photoPrompt,
             texture
        })
    };
    var chatRequest = new ChatRequest(messages, model: "gpt-4-vision-preview");
    var response = await apiVision.ChatEndpoint.GetCompletionAsync(chatRequest);

This generated no error, the image was processed, however the response was cut off.
So I changed my code to ask for response in JSON format and to specify the enum formatting (ChatResponseFormat) to be JSON -

var apiVision = new OpenAIClient(openaikey);
var messages = new List<OpenAI.Chat.Message>
{
    new OpenAI.Chat.Message(Role.System, "You are a helpful assistant designed to output JSON."),
    new OpenAI.Chat.Message(Role.User, new List<OpenAI.Chat.Content>
    {
        photoPrompt,
        texture 
    })
};

var chatRequest = new ChatRequest(messages, "gpt-4-1106-preview", responseFormat: ChatResponseFormat.Json);
//var chatRequest = new ChatRequest(messages, model: "gpt-4-vision-preview");
var response = await apiVision.ChatEndpoint.GetCompletionAsync(chatRequest);`

Expected behavior

I don't understand the error as I'm not passing in an image url (although I have tried it).

Screenshots

Additional context

In summary - The image was accepted and a response received when not specifying Json, but had unending whitespace in response and response was truncated. If I specify Json the api call fails with 'Invalid content type. image_url is only supported by certain models'.

I've also tried this too -

string model = "gpt-4-vision-preview";
ChatResponseFormat responseFormat = ChatResponseFormat.JSON; 

var chatRequest = new ChatRequest(
    messages: messages,
    tools: null, // or your tools if you have any
    model: model,
    responseFormat: responseFormat
);

var response = await apiVision.ChatEndpoint.GetCompletionAsync(chatRequest);`

[StephenHodgson]

Yeah this is pretty strange. I would try to do this in the OpenAI Playground as a start to see if it is even possible.

It could be that the vision-preview model just doesn't support BOTH vision and JSON output modes.

[StephenHodgson]

@GitHubUsername1 https://platform.openai.com/docs/api-reference/chat/create#chat-create-response_format

An object specifying the format that the model must output. Compatible with gpt-4-1106-preview and gpt-3.5-turbo-1106.

[StephenHodgson]

I don't understand the error as I'm not passing in an image url (although I have tried it).

You are passing an image url, since that texture input is being encoded to base64, it is using ImageUrl base type under the hood.

[StephenHodgson]

Have you tried the same code without the JSON formatting?

[GitHubUsername1]

Yes I've tried it without the JSON formatting and with (please see the comments/code above in the Reproduce section). I don't know how I can try it in the Playground, I can't see anywhere to upload an image with Chat and Assistants won't analyze an image. I've tried it with GPT4 web interface as I'm a plus subscriber and the image is accepted and analyzed.

[StephenHodgson]

I would ask around in the OpenAI forum. I don't believe you can use JSON format mode with this model

[GitHubUsername1]

May I add, the reason I tried JSON formatting was because I read this in the Readme in the Chat JSON mode section Chat Json Mode:

"When using JSON mode, always instruct the model to produce JSON via some message in the conversation, for example via your system message. If you don't include an explicit instruction to generate JSON, the model may generate an unending stream of whitespace and the request may run continually until it reaches the token limit. To help ensure you don't forget, the API will throw an error if the string "JSON" does not appear somewhere in the context.
The JSON in the message the model returns may be partial (i.e. cut off) if finish_reason is length, which indicates the generation exceeded max_tokens or the conversation exceeded the token limit. To guard against this, check finish_reason before parsing the response.
"
The response I'm getting to my api call (without JSON), the message in the response is getting cut off and the finish reason is length.

Btw please forgive this noob question but I think this is not using JSON :

var apiVision = new OpenAIClient(openaikey);

var messages = new List<OpenAI.Chat.Message> {
    new OpenAI.Chat.Message(Role.System, "You are a helpful assistant."),
    new OpenAI.Chat.Message(Role.User, new List<OpenAI.Chat.Content> {
        photoPrompt, texture })
};

This is what I'm using.

[GitHubUsername1]

Hmm interesting, I've copied and pasted this code to test from the Readme Chat Vision and the response message is being cut off and finish reason is length, here's the code :

var apiVision = new OpenAIClient(openaikey);

//var messages = new List<OpenAI.Chat.Message> {
//    new OpenAI.Chat.Message(Role.System, "You are a helpful assistant."),
//    new OpenAI.Chat.Message(Role.User, new List<OpenAI.Chat.Content> {
//        photoPrompt, texture })
//};

var messages = new List<OpenAI.Chat.Message> {
    new OpenAI.Chat.Message(Role.System, "You are a helpful assistant."),
    new OpenAI.Chat.Message(Role.User, new List<OpenAI.Chat.Content>{
        "What's in this image?",
         new ImageUrl("https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg", ImageDetail.Low)
    })
};

var chatRequest = new ChatRequest(messages, model: "gpt-4-vision-preview"); 
var response = await apiVision.ChatEndpoint.GetCompletionAsync(chatRequest);

[StephenHodgson]

change your maxTokens to be higher

[GitHubUsername1]

You're a star, that solved it, thankyou!