Ollama Support

[k2an]

I love your project, I want to use it with local ollama+llava and i tried many way including asking chat gpt.
I am on Windows 11, i tried docker and no go. changed api address from settings in frontend also

api key as "ollama" and 
Api Url as http://localhost:11434/v1/ 

and i tested my local ollama+llava answering and running with postman.

changed frontend\src\lib\models.ts

// Keep in sync with backend (llm.py)
// Order here matches dropdown order
export enum CodeGenerationModel {
  GPT_4O_2024_05_13 = "llava", --> HERE
  GPT_4_TURBO_2024_04_09 = "gpt-4-turbo-2024-04-09",
  GPT_4_VISION = "gpt_4_vision",
  CLAUDE_3_SONNET = "claude_3_sonnet",
}

// Will generate a static error if a model in the enum above is not in the descriptions
export const CODE_GENERATION_MODEL_DESCRIPTIONS: {
  [key in CodeGenerationModel]: { name: string; inBeta: boolean };
} = {
  "llava": { name: "LLava", inBeta: false }, --> and here
  "gpt-4-turbo-2024-04-09": { name: "GPT-4 Turbo (Apr 2024)", inBeta: false },
  gpt_4_vision: { name: "GPT-4 Vision (Nov 2023)", inBeta: false },
  claude_3_sonnet: { name: "Claude 3 Sonnet", inBeta: false },
};

also backend\llm.py

Actual model versions that are passed to the LLMs and stored in our logs

class Llm(Enum):
    GPT_4_VISION = "gpt-4-vision-preview"
    GPT_4_TURBO_2024_04_09 = "gpt-4-turbo-2024-04-09"
    GPT_4O_2024_05_13 = "llava"
    CLAUDE_3_SONNET = "claude-3-sonnet-20240229"
    CLAUDE_3_OPUS = "claude-3-opus-20240229"
    CLAUDE_3_HAIKU = "claude-3-haiku-20240307"


 Will throw errors if you send a garbage string
def convert_frontend_str_to_llm(frontend_str: str) -> Llm:
    if frontend_str == "gpt_4_vision":
        return Llm.GPT_4_VISION
    elif frontend_str == "claude_3_sonnet":
        return Llm.CLAUDE_3_SONNET
    elif frontend_str == "llava":
        return Llm.GPT_4O_2024_05_13
    else:
        return Llm(frontend_str)

console and backend errors below

INFO:     ('127.0.0.1', 57364) - "WebSocket /generate-code" [accepted]
Incoming websocket connection...        
INFO:     connection open
Received params
Generating html_tailwind code for uploaded image using Llm.GPT_4O_2024_05_13 model...
Using OpenAI API key from client-side settings dialog
Using OpenAI Base URL from client-side settings dialog
generating code...
ERROR:    Exception in ASGI application
Traceback (most recent call last):      
  File "C:\Users\k\AppData\Local\pypoetry\Cache\virtualenvs\backend-bYKjg4sG-py3.11\Lib\site-packages\uvicorn\protocols\websockets\websockets_impl.py", line 250, in run_asgi
    result = await self.app(self.scope, self.asgi_receive, self.asgi_send)      
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^      
  File "C:\Users\k\AppData\Local\pypoetry\Cache\virtualenvs\backend-bYKjg4sG-py3.11\Lib\site-packages\uvicorn\middleware\proxy_headers.py", line 84, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\k\AppData\Local\pypoetry\Cache\virtualenvs\backend-bYKjg4sG-py3.11\Lib\site-packages\fastapi\applications.py", line 276, in __call__
    await super().__call__(scope, receive, send)
  File "C:\Users\k\AppData\Local\pypoetry\Cache\virtualenvs\backend-bYKjg4sG-py3.11\Lib\site-packages\starlette\applications.py", line 122, in __call__
    await self.middleware_stack(scope, receive, send)
  File "C:\Users\k\AppData\Local\pypoetry\Cache\virtualenvs\backend-bYKjg4sG-py3.11\Lib\site-packages\starlette\middleware\errors.py", line 149, in __call__    
    await self.app(scope, receive, send)
  File "C:\Users\k\AppData\Local\pypoetry\Cache\virtualenvs\backend-bYKjg4sG-py3.11\Lib\site-packages\starlette\middleware\cors.py", line 75, in __call__       
    await self.app(scope, receive, send)
  File "C:\Users\k\AppData\Local\pypoetry\Cache\virtualenvs\backend-bYKjg4sG-py3.11\Lib\site-packages\starlette\middleware\exceptions.py", line 79, in __call__ 
    raise exc
  File "C:\Users\k\AppData\Local\pypoetry\Cache\virtualenvs\backend-bYKjg4sG-py3.11\Lib\site-packages\starlette\middleware\exceptions.py", line 68, in __call__ 
    await self.app(scope, receive, sender)
  File "C:\Users\k\AppData\Local\pypoetry\Cache\virtualenvs\backend-bYKjg4sG-py3.11\Lib\site-packages\fastapi\middleware\asyncexitstack.py", line 21, in __call__
    raise e
  File "C:\Users\k\AppData\Local\pypoetry\Cache\virtualenvs\backend-bYKjg4sG-py3.11\Lib\site-packages\fastapi\middleware\asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "C:\Users\k\AppData\Local\pypoetry\Cache\virtualenvs\backend-bYKjg4sG-py3.11\Lib\site-packages\starlette\routing.py", line 718, in __call__
    await route.handle(scope, receive, send)
  File "C:\Users\k\AppData\Local\pypoetry\Cache\virtualenvs\backend-bYKjg4sG-py3.11\Lib\site-packages\starlette\routing.py", line 341, in handle
    await self.app(scope, receive, send)
  File "C:\Users\k\AppData\Local\pypoetry\Cache\virtualenvs\backend-bYKjg4sG-py3.11\Lib\site-packages\starlette\routing.py", line 82, in app
    await func(session)
  File "C:\Users\k\AppData\Local\pypoetry\Cache\virtualenvs\backend-bYKjg4sG-py3.11\Lib\site-packages\fastapi\routing.py", line 289, in app
    await dependant.call(**values)      
  File "C:\Users\k\screenshot-to-code\backend\routes\generate_code.py", line 262, in stream_code
    completion = await stream_openai_response(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\k\screenshot-to-code\backend\llm.py", line 60, in stream_openai_response
    stream = await client.chat.completions.create(**params)  # type: ignore     
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\k\AppData\Local\pypoetry\Cache\virtualenvs\backend-bYKjg4sG-py3.11\Lib\site-packages\openai\resources\chat\completions.py", line 1334, in create
    return await self._post(
           ^^^^^^^^^^^^^^^^^
y\Cache\virtualenvs\backend-bYKjg4sG-py3.11\Lib\site-packages\openai\_base_client.py", line 1532, in _request
    raise self._make_status_error_from_response(err.response) from None
openai.BadRequestError: Error code: 400 - {'error': {'message': 'json: cannot unmarshal array into Go struct field Message.messages.content of type string', 'type': 'invalid_request_error', 'param': None, 'code': None}}
INFO:     connection closed

If can be use on local server it'll be awesome!
Thanks for consideration

[abi]

We're definitely interested in adding Ollama support to this project. Thanks for opening this issue.

[isaganijaen]

I'm also looking forward to this feature! ✨

[cognitivetech]

👀

[HuangKaibo2017]

Yeah, it will would great to support ollama, LM studio, llama.cpp and more well-known opensource LLMs, like MiniCPM for vision.

[Yitianw]

[immmor]

how's it going now?

[DStarEpoch]

👀

[pdufour]

Ollama already has a OpenAI-compatible API, so all you have to do is change the env variable and create model aliases.

Add to backend/.env

export OPENAI_BASE_URL="http://localhost:11434/v1"
export OPENAI_API_KEY=fake-key

Then create aliases for the models that screenshot-to-code-uses

for model in \
   claude-3-5-sonnet-20240620 \
   gpt-4o-2024-05-13 \
   gpt-4-turbo-2024-04-09 \
   gpt_4_vision \
   claude_3_sonnet; do
   ollama cp x/llama3.2-vision $model
done

Then it will call your local models when it makes LLM requests. Change x/llama3.2-vision to whatever model you want.

I've not gotten very good success because all I can run is a 13b model but perhaps the more powerful models will work well.

Note: you need 0.4.0 release of ollama to run the llama-vision model https://medium.com/@tapanbabbar/how-to-run-llama-3-2-vision-on-ollama-a-game-changer-for-edge-ai-80cb0e8d8928.

[abi]

@pdufour thanks for the tip! I'll include this on the README.

[immmor]

export OPENAI_BASE_URL="http://localhost:11434/v1"
export OPENAI_API_KEY=fake-key

Got this error:
openai.InternalServerError: Error code: 500 - {'error': {'message': 'llama runner process has terminated: error:Missing required key: clip.has_text_encoder', 'type': 'api_error', 'param': None, 'code': None}}

[pdufour]

@immmor see ollama/ollama#7300

[immmor]

ollama/ollama#7300

Thought my ollama is already 0.4.0... Still 0.3.14...

[immmor]

@immmor see ollama/ollama#7300

I tried this model: minicpm-v:latest
It works!

[asifbg-systems]

ollama/ollama#7300

Thought my ollama is already 0.4.0... Still 0.3.14...
You can download pre release
https://github.com/ollama/ollama/releases/tag/v0.4.0-rc8

[davidribs]

Just to clarify for the script kiddies out there. The "for model in" part is not able to be used in the windows command promt. I've only figured it out for powershell. So, open powershell.exe and paste this in:

$models = @(
"claude-3-5-sonnet-20240620",
"gpt-4o-2024-05-13",
"gpt-4-turbo-2024-04-09",
"gpt_4_vision",
"claude_3_sonnet"
)
foreach ($model in $models) {
ollama cp x/llama3.2-vision $model
}

[waiwong614]

How does ollama cp works if I'm running ollama in docker vs ollama installed on host?
Right now, I have ollama docker setup on different PC on the same network.