Connecting LibreChat to a local LM Studio, which hosts a Mistral 7B.

[Angersrezo]

Hello,

I've set up LibreChat to connect to LM Studio, which hosts a Mistral 7B.
Visibly, the whole installation works, on the LM Studio side with the server activated, LibreChat seems OK as it is accessible.
I then try to modify LibreChat's configuration to connect it to my LM Studio server by changing the configuration :

In docker-compose.override.yml, I uncomment the following lines:
services:
api:
volumes:
- ./librechat.yaml:/app/librechat.yaml

In librechat.yaml I point to my local server, specifying the parameters provided by my LMStudio server:
- name: "Mistral" # Unique name for the endpoint
apiKey: "not-needed"
baseURL: http://localhost:1234/v1

I launch Docker : docker-compose up

This is the error returned:

PS C:\LibreChat> docker-compose up
time="2024-02-19T15:13:11+01:00" level=warning msg="The "UID" variable is not set. Defaulting to a blank string."
time="2024-02-19T15:13:11+01:00" level=warning msg="The "GID" variable is not set. Defaulting to a blank string."
time="2024-02-19T15:13:11+01:00" level=warning msg="The "UID" variable is not set. Defaulting to a blank string."
time="2024-02-19T15:13:11+01:00" level=warning msg="The "GID" variable is not set. Defaulting to a blank string."
time="2024-02-19T15:13:11+01:00" level=warning msg="The "UID" variable is not set. Defaulting to a blank string."
time="2024-02-19T15:13:11+01:00" level=warning msg="The "GID" variable is not set. Defaulting to a blank string."
yaml: line 15: did not find expected key
PS C:\LibreChat>

Has anyone ever tried to connect to an LM Studio server, especially with a mistral AI?

Vincent.

[danny-avila]

I don’t think this is specific to LMStudio but you may have an incorrect configuration of the librechat.yaml file or the compose file if you made edits there:

yaml: line 15: did not find expected key

share either here with sensitive values removed. You can use https://yamlchecker.com/ to verify

also check your logs in ./logs, the latest debug* and error* logs are both relevant for this

[Angersrezo]

Hi Danny, and thanks for coming back to me.

https://yamlchecker.com/ did not detect any errors.

The librechat.yaml:

# # For more information, see the Configuration Guide:

# https://docs.librechat.ai/install/configuration/custom_config.html

# Configuration version (required)
version: 1.0.3

# Cache settings: Set to true to enable caching
cache: true

# Example Registration Object Structure (optional)
registration:
  socialLogins: ["github", "google", "discord", "openid", "facebook"]
  # allowedDomains:
    # - "gmail.com"

# fileConfig:
#   endpoints:
#     assistants:
#       fileLimit: 5
#       fileSizeLimit: 10  # Maximum size for an individual file in MB
#       totalSizeLimit: 50  # Maximum total size for all files in a single request in MB
#       supportedMimeTypes:
#         - "image/.*"
#         - "application/pdf"
#     openAI:
#       disabled: true  # Disables file uploading to the OpenAI endpoint
#     default:
#       totalSizeLimit: 20
#     YourCustomEndpointName:
#       fileLimit: 2
#       fileSizeLimit: 5
#   serverFileSizeLimit: 100  # Global server file size limit in MB
#   avatarSizeLimit: 2  # Limit for user avatar image size in MB
# rateLimits:
#   fileUploads:
#     ipMax: 100
#     ipWindowInMinutes: 60  # Rate limit window for file uploads per IP
#     userMax: 50
#     userWindowInMinutes: 60  # Rate limit window for file uploads per user

# Definition of custom endpoints
endpoints:
  # assistants:
  #   disableBuilder: false # Disable Assistants Builder Interface by setting to `true`
  #   pollIntervalMs: 750  # Polling interval for checking assistant updates
  #   timeoutMs: 180000  # Timeout for assistant operations
  #   # Should only be one or the other, either `supportedIds` or `excludedIds`
  #   supportedIds: ["asst_supportedAssistantId1", "asst_supportedAssistantId2"]
  #   # excludedIds: ["asst_excludedAssistantId"]
  custom:
    # Mistral AI API
    - name: "Mistral"  # Unique name for the endpoint
      # For `apiKey` and `baseURL`, you can use environment variables that you define.
      # recommended environment variables:
      apiKey: "not-needed"
      baseURL: "http://127.0.0.1:1234/v1"

      # Models configuration
      models: 
        # List of default models to use. At least one value is required.
        default: ["mistral-tiny", "mistral-small", "mistral-medium"]
        # Fetch option: Set to true to fetch models from API.
        fetch: true  # Defaults to false.

      # Optional configurations
      
      # Title Conversation setting
      titleConvo: true  # Set to true to enable title conversation

      # Title Method: Choose between "completion" or "functions".
      titleMethod: "completion"  # Defaults to "completion" if omitted.

      # Title Model: Specify the model to use for titles.
      titleModel: "mistral-tiny"  # Defaults to "gpt-3.5-turbo" if omitted.

      # Summarize setting: Set to true to enable summarization.
      summarize: false

      # Summary Model: Specify the model to use if summarization is enabled.
      summaryModel: "mistral-tiny"  # Defaults to "gpt-3.5-turbo" if omitted.

      # Force Prompt setting: If true, sends a `prompt` parameter instead of `messages`.
      forcePrompt: false

      # The label displayed for the AI model in messages.
      modelDisplayLabel: "Mistral"  # Default is "AI" when not set.

      # Add additional parameters to the request. Default params will be overwritten.
      addParams:
        safe_prompt: false # This field is specific to Mistral AI: https://docs.mistral.ai/api/
        
      # Drop Default params parameters from the request. See default params in guide linked below.
      # NOTE: For Mistral, it is necessary to drop the following parameters or you will encounter a 422 Error:
      dropParams: ["stop", "user", "frequency_penalty", "presence_penalty"]
      
    # OpenRouter.ai Example
    - name: "OpenRouter"
      # For `apiKey` and `baseURL`, you can use environment variables that you define.
      # recommended environment variables:
      # Known issue: you should not use `OPENROUTER_API_KEY` as it will then override the `openAI` endpoint to use OpenRouter as well.
      apiKey: "${OPENROUTER_KEY}"
      baseURL: "https://openrouter.ai/api/v1"
      models:
        default: ["gpt-3.5-turbo"]
        fetch: true
      titleConvo: true
      titleModel: "gpt-3.5-turbo"
      summarize: false
      summaryModel: "gpt-3.5-turbo"
      forcePrompt: false
      modelDisplayLabel: "OpenRouter"

# See the Custom Configuration Guide for more information:
# 

> https://docs.librechat.ai/install/configuration/custom_config.html

docker-compose.override.yml:

version: '3.4'

# Please consult our docs for more info: https://docs.librechat.ai/install/configuration/docker_override.html

# TO USE THIS FILE, FIRST UNCOMMENT THE LINE ('services:')

# THEN UNCOMMENT ONLY THE SECTION OR SECTIONS CONTAINING THE CHANGES YOU WANT TO APPLY
# SAVE THIS FILE AS 'docker-compose.override.yaml'
# AND USE THE 'docker compose build' & 'docker compose up -d' COMMANDS AS YOU WOULD NORMALLY DO

# WARNING: YOU CAN ONLY SPECIFY EVERY SERVICE NAME ONCE (api, mongodb, meilisearch, ...)
# IF YOU WANT TO OVERRIDE MULTIPLE SETTINGS IN ONE SERVICE YOU WILL HAVE TO EDIT ACCORDINGLY

# EXAMPLE: if you want to use the config file and the latest docker image the result will be:

 services:
   api:
     volumes:
       - ./librechat.yaml:/app/librechat.yaml
#     image: ghcr.io/danny-avila/librechat-dev:latest

# ---------------------------------------------------

# services:

# # USE LIBRECHAT CONFIG FILE
#   api:
#     volumes:
#       - ./librechat.yaml:/app/librechat.yaml

# # BUILD FROM LATEST IMAGE
#   api:
#     image: ghcr.io/danny-avila/librechat-dev:latest
 
# # BUILD FROM LATEST IMAGE (NUMBERED RELEASE)
#   api:
#     image: ghcr.io/danny-avila/librechat:latest

# # BUILD FROM LATEST API IMAGE
#   api:
#     image: ghcr.io/danny-avila/librechat-dev-api:latest

# # BUILD FROM LATEST API IMAGE (NUMBERED RELEASE)
#   api:
#     image: ghcr.io/danny-avila/librechat-api:latest

# # ADD MONGO-EXPRESS
#   mongo-express:
#     image: mongo-express
#     container_name: mongo-express
#     environment:
#       ME_CONFIG_MONGODB_SERVER: mongodb
#       ME_CONFIG_BASICAUTH_USERNAME: admin
#       ME_CONFIG_BASICAUTH_PASSWORD: password
#     ports:
#       - '8081:8081'
#     depends_on:
#       - mongodb
#     restart: always

# # USE MONGODB V4.4.18 - FOR OLDER CPU WITHOUT AVX SUPPORT
#   mongodb:
#     image: mongo:4.4.18

# # DISABLE THE MONGODB CONTAINER - YOU NEED TO SET AN ALTERNATIVE MONGODB URI IN THE .ENV FILE
#   api:
#     environment:
#       - MONGO_URI=${MONGO_URI}
#   mongodb:
#     image: tianon/true
#     command: ""
#     entrypoint: ""

# # EXPOSE MONGODB PORTS - USE CAREFULLY, THIS MAKES YOUR DATABASE VULNERABLE TO ATTACKS
#   mongodb:
#     ports:
#       - 27018:27017

# # DISABLE MEILISEARCH
#   meilisearch:
#     profiles:
#       - donotstart

# # EXPOSE MEILISEARCH PORTS - DO NOT USE THE DEFAULT VALUE FOR THE MASTER KEY IF YOU DO THIS
#   meilisearch:
#     ports:
#       - 7700:7700

[danny-avila]

https://yamlchecker.com/ did not detect any errors.

Your override file does generate an error for me as is:

Try this:

version: '3.4'

services:

# USE LIBRECHAT CONFIG FILE
  api:
    volumes:
      - ./librechat.yaml:/app/librechat.yaml

[Angersrezo]

Thanks,

Disably, it launches.
There's no discussion with my server after that, but I'll have a look.

[Angersrezo]

Here's the message returned when I tried {"level":"error","message":"Failed to fetch models from Mistral API Something happened in setting up the request Cannot read properties of undefined (reading 'status')"}

[fuegovic]

This is what I used in my librechat.yaml for Mistral in LMStudio:

    # Mistral AI API
    - name: "Mistral 7B"
      apiKey: "sk-1234"
      baseURL: "http://localhost:1234/v1"
      models: 
        default: [
          "local-model",
          ]
        fetch: false
      titleConvo: true
      titleMethod: "completion"
      titleModel: "local-model"
      summarize: false
      summaryModel: "local-model"
      forcePrompt: false
      modelDisplayLabel: "Mistral 7B"
      dropParams: ["stop", "user", "frequency_penalty", "presence_penalty"]

Note: in LMStudio you need to enable the "server mode" since it's disabled by default

Edit: removed "user_provided" and replaced it with a dummy key

[Angersrezo]

I have use your setup, and it's not better for me.
I had to enter some kind of API key to access the input field on LibreChat. It seems to me that the API key isn't necessary anyway with LM Studio local.

It would be very useful for me to have access to it, as there are several of us who need it.
Fuegovic, do you have the possibility to send me your LibreChat file, so that I can test it?

[fuegovic]

Note: I edited my example above to fix this

My bad, I've left it set as "user_provided". You just have to enter a dummy key to fix this, it can be anything. "mistral", "1234", "some-random-string-of-characters", etc. You can either enter the dummy key in the UI (look for the cog icon in the endpoint menu) or enter it in the librechat.yaml, at this line (replace "user_provided" with the dummy key):

apiKey: "sk-1234"

[Angersrezo]

Top, thank you very much Fuegovic,

And do you have any idea why I can't connect to my ML Studio server?
I can't get rid of the error message:

[Angersrezo]

I have this message on my powershell

[fuegovic]

are you able to manually do a (chat) request to LMStudio server with curl or python using their examples?

[Angersrezo]

Yes, I did the test and it works.
Contrary to what I find elsewhere, I'm obliged to use a line to specify an Api code for it to work, even if it's not used.

[badcom]

I'm getting the same error when connecting to Mistral via CloudFlare:

2024-02-22 07:56:17 error: [OpenAIClient.chatCompletion] Unhandled error type Error: Connection error.
2024-02-22 07:56:17 error: [handleAbortError] AI response error; aborting request: Error: Connection error.

Debug logs:

2024-02-22 07:56:15 debug: [OpenAIClient] chatCompletion
{
  baseURL: "https://api.cloudflare.com/client/v4/accounts/[my_account]/ai/run/@cf/mistral/mi... [truncated]",
    modelOptions.model: "mistral-tiny",
    modelOptions.temperature: 1,
    modelOptions.top_p: 1,
    modelOptions.presence_penalty: 0,
    modelOptions.frequency_penalty: 0,
    // 3 stop(s)
    modelOptions.stop: ["||>","\nUser:","<|diff_marker|>"],
    modelOptions.user: "65b4ef76846b7fa5fc808a77",
    modelOptions.stream: true,
    // 1 message(s)
    modelOptions.messages: [{"role":"user","content":"Hey there"}],
}
2024-02-22 07:56:17 warn: [OpenAIClient.chatCompletion][stream] API error
2024-02-22 07:56:17 warn: [OpenAIClient.chatCompletion][finalChatCompletion] API error
2024-02-22 07:56:17 error: [OpenAIClient.chatCompletion] Unhandled error type Error: Connection error.
2024-02-22 07:56:17 error: [handleAbortError] AI response error; aborting request: Error: Connection error.

Error logs:

{"level":"error","message":"[OpenAIClient.chatCompletion] Unhandled error type Error: Connection error.","stack":"Error: Error: Connection error.\n    at OpenAIClient.chatCompletion (C:\\laragon\\www\\librechat-private\\api\\app\\clients\\OpenAIClient.js:1099:15)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async OpenAIClient.sendCompletion (C:\\laragon\\www\\librechat-private\\api\\app\\clients\\OpenAIClient.js:598:15)\n    at async OpenAIClient.sendMessage (C:\\laragon\\www\\librechat-private\\api\\app\\clients\\BaseClient.js:449:24)\n    at async AskController (C:\\laragon\\www\\librechat-private\\api\\server\\controllers\\AskController.js:111:20)\n    at async C:\\laragon\\www\\librechat-private\\api\\server\\routes\\ask\\custom.js:24:5"}
{"level":"error","message":"[handleAbortError] AI response error; aborting request: Error: Connection error.","stack":"Error: Error: Connection error.\n    at OpenAIClient.chatCompletion (C:\\laragon\\www\\librechat-private\\api\\app\\clients\\OpenAIClient.js:1099:15)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async OpenAIClient.sendCompletion (C:\\laragon\\www\\librechat-private\\api\\app\\clients\\OpenAIClient.js:598:15)\n    at async OpenAIClient.sendMessage (C:\\laragon\\www\\librechat-private\\api\\app\\clients\\BaseClient.js:449:24)\n    at async AskController (C:\\laragon\\www\\librechat-private\\api\\server\\controllers\\AskController.js:111:20)\n    at async C:\\laragon\\www\\librechat-private\\api\\server\\routes\\ask\\custom.js:24:5"}

[badcom]

Nevermind. Disabling my VPN fixed this.

[Angersrezo]

Anyone have any idea why this isn't working? Really impossible to connect to my LM Studio with my LibreChat.

[danny-avila]

The issue is the connection.

I noticed a discrepancy between your python script and the custom config:

  baseURL: "http://127.0.0.1:1234/v1"

you have the ip listed as 127.0.0.1, whereas in your script it's localhost. Could be worth a shot. You could also try 0.0.0.0

@fuegovic uses docker and uses localhost

[Angersrezo]

Hi, Danny,
That doesn't change anything.
It seems to be a communication problem with or through the API.
Is it possible to check that I have the right version?

[Angersrezo]

I give up for now, as I can't find any solution.
I'll try to connect to Perplexity.

[fuegovic]

If you are still interested in adding Perplexity to LibreChat, we recently documented it:
https://docs.librechat.ai/install/configuration/ai_endpoints.html#perplexity

[Angersrezo]

A quick update on my problem.
Apparently, all my problems stemmed from characters not visible in my yaml and yml configuration files.
Strangely enough, the ones I downloaded with the git clone already had the problem, but I was able to retrieve them elsewhere t cala solved my problems.
I've been able to get most of the features working, api, user account management...
Great work Danny, your project is really impressive. I'm trying to get my manager to let me deploy it on our sites.

Now I've got to connect it to a local Mistral AI on an LM Studio server, but obviously that's not possible yet, as I haven't found anyone who has.

The next step would be to run a training session on this AI.

[fuegovic]

Now I've got to connect it to a local Mistral AI on an LM Studio server, but obviously that's not possible yet, as I haven't found anyone who has.

It is possible to do it, here's the exact config file I used when I screen recorded this:
(both LibreChat and LMStudio were fully updated, and I ran LibreChat with npm instead of docker this time, but both are working)

If you still can't get it to work, as an alternative you can use Ollama: https://docs.librechat.ai/install/configuration/ollama.html

mistral-local.mp4

version: 1.0.4

cache: false

endpoints:
  custom:
    # Mistral AI API
    - name: "Mistral 7B"
      apiKey: "sk-1234"
      baseURL: "http://localhost:1234/v1"
      models: 
        default: [
          "local-model",
          ]
        fetch: false
      titleConvo: true
      titleMethod: "completion"
      titleModel: "local-model"
      summarize: false
      summaryModel: "local-model"
      forcePrompt: false
      modelDisplayLabel: "Mistral 7B"
      dropParams: ["stop", "user", "frequency_penalty", "presence_penalty"]

[Angersrezo]

Still can't connect to my LM Studio server.
I've replaced my yaml with yours to avoid the risk of hidden characters.
Fearing it might be an access problem, I've removed my antivirus and shut down my firewall, but my Mistral is still unreachable.

https://www.youtube.com/watch?v=YwKGwOeCGcQ

[Angersrezo]

@fuegovic , Knowing that I can connect to my LM studio with Python, could it be that ports like 1234 are not open on docker?
Because there's no doubt that everything works, but only the communication between Lm Studio and LibreChat isn't working.

[fuegovic]

@Angersrezo I think I know the problem, for some reason I thought you were running the local install (aka with npm) instead of docker (or I forgot you were using docker). Docker has its own network, so localhost will point to the wrong place (inside docker). Try setting base url like this instead: baseURL: "http://host.docker.internal:1234/v1"

Using host.docker.internal should allow communication to your host pc localhost address

Apologies for the oversight! 😊

Just tested with docker on a fresh install and it worked

version: 1.0.4

cache: false

endpoints:
  custom:
    # Mistral AI API
    - name: "Mistral 7B"
      apiKey: "sk-1234"
      baseURL: "http://host.docker.internal:1234/v1"
      models: 
        default: [
          "local-model",
          ]
        fetch: false
      titleConvo: true
      titleMethod: "completion"
      titleModel: "local-model"
      summarize: false
      summaryModel: "local-model"
      forcePrompt: false
      modelDisplayLabel: "Mistral 7B"
      dropParams: ["stop", "user", "frequency_penalty", "presence_penalty"]

[Angersrezo]

I think you must be close to the problem. Because once again I was disappointed when I saw that it didn't work.
I completely redid my installation, even reinstalling GIT and all the applications, but the problem didn't go away.
I tested on 3 of my machines, with non-domain machines, without antivirus, etc... nothing helped.
At one point I said to myself that since I can reach all the external AIs, but not the internal ones, that means everything's working and you're right, it's due to the fact that I'm local.
So I logged on with one of my machines on another, and now, finally, it's responding.

Since I'm going to dedicate a server to hosting LibreChat, this won't be a problem for me. However, it will be difficult to test locally.
It would still be interesting to find out how to do this locally, because even with my IP on the network, it doesn't work.

Thanks a lot for your support @fuegovic

[fuegovic]

Using your local network IP was my alternate solution... All I can think right now to enable you to test locally is more a workaround than a solution. You could use ngrok or cloudflare tunnels to make localhost:1234 accessible from the web, and then use this external connection to connect back to LMStudio.

[fredeq]

Not sure if it's still needed but I solved the exact same problem by changing 127.0.0.1 to host.docker.internal (=> baseURL: 'http://host.docker.internal:1234/v1')