Error When Accessing Azure Serverless Models

[MasterOfWebM]

What happened?

When attempting to utilize a serverless model in Azure AI Foundry (Azure OpenAI), I am receiving a 401 error, even though the the API key is verified and correct.

Config:

endpoints:
  azureOpenAI:
    titleModel: 'Meta-Llama-3.1-8B-Instruct'
    plugins: false
    assistants: false
    groups:
    - group: 'Llama'
      serverless: true
      apiKey: '<REDACTED>'
      baseURL: 'https://<REDACTED>.services.ai.azure.com/models/chat/completions?api-version=2024-05-01-preview'
      models:
        Meta-Llama-3.1-8B-Instruct: true

I tried the baseURL with and without the query parameter without any luck.

Steps to Reproduce

1. Create a new serverless model in Azure AI Foundry
2. Copy the Target URI in Foundry and use it as the baseURL
3. Attempt to query the API endpoint within LibreChat

What browsers are you seeing the problem on?

No response

Relevant log output

error: [handleAbortError] AI response error; aborting request: Failed to send message. HTTP 401 - { "statusCode": 401, "message": "Unauthorized. Access token is missing, invalid, audience is incorrect (https://cognitiveservices.azure.com), or have expired." }

Screenshots

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[danny-avila]

I'm having trouble with Azure serverless models even with raw (cURL) API requests, including all the code examples that Azure gives

[danny-avila]

Figured it out after running a couple of different tests. All their documentations say to use Auth Bearer for serverless inference requests (source):

POST /chat/completions?api-version=2024-04-01-preview
Authorization: Bearer <bearer-token>
Content-Type: application/json

however, we need to use api-key header instead:

I will make some changes to account for this, as well as supporting models/chat/completions as opposed to legacy /v1/chat/completions

[MasterOfWebM]

Much appreciated!

[danny-avila]

I'm updating the docs, but the update is now live:

Here is an example configuration for Meta-Llama-3.1-8B-Instruct:

endpoints:
  azureOpenAI:
    groups:
    - group: "serverless-example"
      apiKey: "${LLAMA318B_API_KEY}"
      baseURL: "https://example.services.ai.azure.com/models/"
      version: "2024-05-01-preview" # Optional: specify API version
      serverless: true
      models:
        # Must match the deployment name of the model
        Meta-Llama-3.1-8B-Instruct: true

Notes:

• Azure AI Foundry models now provision endpoints under /models/chat/completions?api-version=version for serverless inference.
  • The baseURL field should be set to the root of the endpoint, without anything after /models/, i.e., the /chat/completions path.
  • Example: https://example.services.ai.azure.com/models/ for https://example.services.ai.azure.com/models/chat/completions?api-version=2024-05-01-preview
  • The version query parameter is optional and can be specified in the baseURL field.
• The model name used in the models field must match the deployment name of the model in the Azure AI Foundry.
• Compatibility with LibreChat relies on parity with OpenAI API specs, which at the time of writing, are typically "Pay-as-you-go" or "Models as a Service" (MaaS) deployments on Azure AI Studio, that are OpenAI-SDK-compatible with either v1/completions or models/chat/completions endpoint handling.
• All models that offer serverless deployments ("Serverless APIs") are compatible from the Azure model catalog. You can filter by "Serverless API" under Deployment options and "Chat completion" under inference tasks to see the full list; however, real time endpoint models have not been tested.
• These serverless inference endpoint/models may or may not support function calling according to OpenAI API specs, which enables their use with Agents.
• If using legacy "/v1/completions" (without "chat"), you need to set the forcePrompt field to true in your group config.