Invoke local.ai API via LangChain and/or OpenAI Python API?

[mstanley103]

I think the title covers the basic question. I've spent quite a bit of time experimenting with LangChain and OpenAI APIs via Python. After getting disgusted with continuing OpenAI timeouts, I've downloaded local.ai and have a model running there. How do I tie that server back into my existing Python code?

[louisgv]

@Regular-Baf people NEED the WIKI hahahah

We have a rolling issue since last week: #30 @mstanley103 you can take a look at that issue for pointer on how to implement the backend - you can prob just use openai API and replace the host name/openai endpoint with localai's localhost:8000/completions

I will check-in with @Regular-Baf (he's the current assignee of this tix atm), will crank it out by TMR. @mstanley103 if you would like to contribute to the wiki LMK, we def need some docs on how to work using python

[mstanley103]

Thanks @louisgv,

I've tried both styles of OpenAI APIs and in both cases I'm receiving "HTTP code 404 from API ()" which I understand to mean that the server has been contacted, but the expected resource could not be found. Here's a short test program that can be used to replicate my results. It can run against either OpenAI or Local.ai just by changing the LOCAL variable. I also tried both styles of API. That choice is controlled by the ChatCompletion variable.

Thanks for the quick response!

Best regards,

@mstanley103

---

import openai

ChatCompletion=True
LOCAL=True

def get_completion(prompt, model="gpt-3.5-turbo"):
    messages = [{"role": "user", "content": prompt}]
    response = openai.ChatCompletion.create(
        model=model,
        messages=messages,
        temperature=0, # this is the degree of randomness of the model's output
    )
    return response.choices[0].message["content"]

if LOCAL:
    model = "GPT4All-J 13B Snoozy"
    openai.api_key = "YOUR_API_KEY"
    openai.api_base = "http://localhost:8000/completions"
else:
    if ChatCompletion:
        model = "gpt-3.5-turbo"
    else:
        model = 'text-davinci-002'

prompt = "What is the capital of Arizona?."

if ChatCompletion:
    try:
        generated_text = get_completion(prompt, model=model)   
        print('via ChatCompletion:', generated_text)
    except Exception as e:
        print('Exception in openai.ChatCompletion.create: ', e)
else:
    try:
        response = openai.Completion.create(engine=model, prompt=prompt, max_tokens=50)
        generated_text = response.choices[0].text
        print('via Completion.create: ', generated_text)
    except Exception as e:
        print('Exception in openai.completion.create: ', e)

[louisgv]

@mstanley103 you have a typo in the Completion.create call (I think):

response = openai.Completion.create(
  model=model,
  prompt=prompt,
  max_tokens=420,
)

Note that there's no chat completion endpoint ATM, only prompt endpoint!

We will see to add a chat-dedicated endpoint down the line once we got #14 and #13 functional :d

[mstanley103]

@louisgv,
You're right that max_tokens was too small and I should have used "model" instead of "engine". But that wasn't the issue. I just fixed both problems and got the same result. Interestingly OpenAI did not complain about either!
I had guessed you might not have ChatCompletions yet. Thanks for the confirmation.
Feel free to keep throwing suggestions to me. I've got the environment up so it's easy to run new ideas.
The updated code is here. I've tacked on a couple notes at the end.

import openai

ChatCompletion=False
LOCAL=True

def get_completion(prompt, model="gpt-3.5-turbo"):
    messages = [{"role": "user", "content": prompt}]
    response = openai.ChatCompletion.create(
        model=model,
        messages=messages,
        temperature=0, # this is the degree of randomness of the model's output
    )
    return response.choices[0].message["content"]

if LOCAL:
    model = "GPT4All-J 13B Snoozy"
    openai.api_key = "YOUR_API_KEY"
    openai.api_base = "http://localhost:8000/completions"
else:
    if ChatCompletion:
        model = "gpt-3.5-turbo"
    else:
        model = 'text-davinci-002'

prompt = "What is the capital of Arizona?."

if ChatCompletion:
    try:
        generated_text = get_completion(prompt, model=model)   
        print('via ChatCompletion:', generated_text)
    except Exception as e:
        print('Exception in openai.ChatCompletion.create: ', e)
else:
    try:
        response = openai.Completion.create(model=model, prompt=prompt, max_tokens=1024)
        generated_text = response.choices[0].text
        print('via Completion.create: ', generated_text)
    except Exception as e:
        print('Exception in openai.completion.create: ', e)

and as before, the error code is:
Exception in openai.completion.create: HTTP code 404 from API ()

I dug a bit more into the exception. Under the hood, it has this structure:
{'_message': 'HTTP code 404 from API ()',
'http_body': '',
'http_status': 404,
'json_body': None,
'headers': {'content-length': '0', 'vary': 'Origin, Access-Control-Request-Method, Access-Control-Request-Headers', 'access-control-allow-credentials': 'true', 'date': 'Thu, 06 Jul 2023 17:26:14 GMT'},
'code': None,
'request_id': None,
'error': None,
'organization': None}

[louisgv]

@mstanley103 try adding stream to it:

openai.Completion.create(model=model, prompt=prompt, max_tokens=1024, stream=true)

I suppose by default their client API assume one-shot response and not a streaming response :-? ... <- will need to add a non-stream response later.

[louisgv]

Also see: https://github.com/louisgv/local.ai/wiki/Using-the-local.ai-API

We should add your python example once it's confirmed working :-?

[mstanley103]

Stay tuned :-)

[mstanley103]

@louisgv,
I have a quick and dirty solution for you that is completely standalone and is a very thin layer on top of what you have documented.

import requests
import json

def get_completion(prompt, max_tokens=512, temperature=0.6):
    global response
    url = 'http://localhost:8000/completions'
    headers = {
        'Content-Type': 'application/json'
    }
    data = {
        "prompt": prompt,
        "max_tokens": max_tokens,
        "temperature": temperature,
    }

    response = requests.post(url, headers=headers, data=json.dumps(data))

    completion = ''
    if response.status_code == 200:
        content = response.content.decode('ISO-8859-1')
        lines = content.split('\n')
        for line in lines:
            if line.startswith('data:'):
                if line.endswith('[DONE]'):
                    break
                else:
                    choice = json.loads(line[6:])
                    txt = choice['choices'][0]['text']
                    completion += txt
        return(completion)
    else:
        return(None)

I have a Jupyter notebook I would be happy to contribute if you would tell me where you would like the file uploaded. I can add a page to your wiki as well if you would like. Just reply here and let me know.
Best regards,
@mstanley103

[louisgv]

@mstanley103 feel free to edit the Wiki as desired! I think either a runnable jupyter notebook or sample code in wiki are all welcome, whichever seems most helpful to future passerby!

[mstanley103]

Wiki page added. Thanks for your help!

[mstanley103]

See new wiki page at API-with-Python.