Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Picking different model between tab complete and chat incurs loading model to memory penalty lag #137

Open
jamescho72 opened this issue Nov 12, 2024 · 4 comments
Assignees

Comments

@jamescho72
Copy link

ollama does not load more than 1 model at a time.
Configuring 2 different models under config.json in continue causes ~45-60 second lag between calls to tab complete vs chat for a poor experience. tab complete appears broken each time it need to re-load into memory.

We may want to enforce single model for both chat and tab code complete until this can be resolved.

@fbricon
Copy link
Collaborator

fbricon commented Nov 14, 2024

The 1st immediate steps would be to:

  • remove the tab completion combo (comment the code)
  • change the "Chat Model" label to "Chat/Tab Completion Model"
  • on the backend side copy the chatModel value into the tabCompletionModel value

That should work.

Then in an another steps/PR:

  • change the UI to add a "Advanced" button
  • on click, it would reveal the tab completion combo, rename the button to "Simple"
  • toggle "Chat/Tab Completion Model" label to "Chat Model"
  • on the backend side, only copy the chatModel value into the tabCompletionModel value if UI mode is advanced

@fbricon
Copy link
Collaborator

fbricon commented Nov 14, 2024

@SachinS10-lab part 1 is done in #139, can you please add the advanced section, where the tab completion can be selected?

@SachinS10-lab
Copy link
Contributor

Added toggle button for advanced section.

Image


For Simple Configuration

Image


For Advanced Configuration

Image


I also add an information message below the box for better user experience. Is this Okay?

Image

@fbricon
Copy link
Collaborator

fbricon commented Nov 18, 2024

For the advanced section, you forgot

on click, it would reveal the tab completion combo,

Please open a PR, it'll be easier to review than commenting on screenshots

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants