Jan: Remote APIs

[dan-homebrew]

Overview

• What is our Remote API architecture for v0.5.3
• What is our plan for Remote API architecture in v0.6? (i.e. Cortex Platform extensions?)

Goal

• We define a clear API for Remote API Extensions
• Remote API Extensions are standalone repos (e.g. groq-extension)
• Remote API Extensions can update their own model lists

Thoughts

• Can we define a clear API for Remote API Extensions?
• Remote API extension should be its own Github Repo, can be maintained by community
• How do we populate remote model lists? (e.g. vs. manually adding)
• [Potentially unsolvable] How do we handle file uploads, voice etc in the future?

[marknguyen1302]

What is our Remote API architecture for v0.5.3?:

• Build time:
  • We have 2 variables that will be defined in the build time: settings and models, it will be passed to the extension via Webpack.
    
  • Settings:
    • Will be registered when Jan loads the extension using registerSettings method from BaseExtension then it will be copied to settings folder in Jan and show on the model provider settings page,
  • Models:
    • For now, all the remote models are static and we add them manually, then they will be registered when Jan loads the extension using the registerModels method from BaseExtension in run time.
    • But in the build time, we can fetch all models from remote providers in the build time (similar to what @louis-jan did in Cortex Extension), by this approach, we can populate remote model lists from remote provider
• Run time:
  • Settings:
    • Setting items will be populated and mapped to the UI by SettingDetailItem(web/screens/Settings/SettingDetail/SettingDetailItem/index.tsx), it supports input,checkbox for now
    • When settings are changed, the updateSettings method in BaseExtension will be called, it will change setting.json and call the onSettingUpdate method of the extension
• Models:
  • The model extension will load all models registered and map with the remote extension by the engine field
  • Inference:
    • We have 3 methods to change the inference to the remote engine, headers, transformPayload, transformResponse
    • headers are for adding additional headers that are required by the remote engine, default headers will have Authorization and api-key. For example, Anthropic will need an anthropic-version header
    • transformPayload is for transforming the payload in Inference payload
    • transformResponse is for transforming the response of the remote engine to a response that is compatible with the OpenAI

• What is our plan for Remote API architecture in v0.6?:
  • I think the plan is still to move all remote extensions to use Cortex Extension since it already supports them.
• Can we define a clear API for Remote API Extensions?
  • I'm sorry but could you explain more about this?
• Remote API extension should be its own Github Repo, which can be maintained by community:
  • I think it's a great idea, I just asked @louis-jan about this, and seems we did something similar before, we can have a category in Jan and it will list all remote extension repositories, and then we can fetch them in build time.
• How do we populate remote model lists?
  • I think I already added this above
• [Potentially unsolvable] How do we handle file uploads, voice etc in the future? (still checking this)
  cc @dan-homebrew @louis-jan

[marknguyen1302]

graph TD

%% Build Time
subgraph Build Time
  A[Settings] --> |Define & Pass via Webpack| B[Extension]
  D[Models] --> |Define & Pass via Webpack| B
  
  B --> |Register using registerSettings, registerModels| C["Copy to Settings, Models folders in Jan"]
  C --> |Show on Model Provider Settings Page and Model Hub| D1[UI]

  D --> |Idea -> Fetch Models from Remote Providers| E[Remote Model Provider]
end

%% Run Time
subgraph Run Time
  F[Extension] --> |Load & Register Settings| G["registerSettings Method"]
  G --> |Populate & Map to UI| H["SettingDetailItem in web/screens/Settings/SettingDetail/SettingDetailItem/index.tsx"]
  H --> |Supports Input, Checkbox| I[UI]
  
  F --> |Load & Register Models| J["registerModels Method"]
  J --> |Map to Remote Extension by Engine Field| K[Model Extension]

  L[Inference] --> |Change Inference| M["headers, transformPayload, transformResponse"]
  M --> |Headers: Authorization, API-key| N[Remote Engine]
  M --> |Additional headers, example: anthropic-version| N
end

%% Connections
A --> |Interact with| G
D --> |Interact with| J

Loading