feat(forge/llm): Add LlamafileProvider

[k8si]

Background

This draft PR is a step toward enabling the use of local models in AutoGPT by adding llamafile as an LLM provider.

Implementation notes are included in forge/forge/llm/providers/llamafile/README.md

Related issues:

• Instructions for local models #6336
• Decouple thoughts generation from function call generation #6947

Depends on:

• refactor(forge/llm): Create BaseOpenAIProvider -> deduplicate GroqProvider & OpenAIProvider #7178
• docs+fix(agent): Update LLM setup instructions & remove GPT-specific flags #7183

Changes 🏗️

• Add minimal implementation of LlamafileProvider, a new ChatModelProvider for llamafiles. It extends BaseOpenAIProvider and only overrides methods that are necessary to get the system to work at a basic level.

• Add support for mistral-7b-instruct-v0.2. This is the only model currently supported by LlamafileProvider because this is the only model I tested anything with.

• Misc changes to app configuration to enable switching between openai/llamafile providers. In particular, added config field LLM_PROVIDER that, when set to 'llamafile', will use LllamafileProvider in agents rather than OpenAIProvider.

• Add instructions to use AutoGPT with llamafile in the docs at autogpt/setup/index.md

Limitations:

• Only tested with (quantized) Mistral-7B-Instruct-v0.2
• Only tested with a single AutoGPT 'task' ("Tell me about Roman dodecahedrons")
• Did not attempt extensive refactoring of existing components; I just added special cases as necessary
• Haven't added any tests for new classes/methods

PR Quality Scorecard ✨

• Have you used the PR description template?   +2 pts
• Is your pull request atomic, focusing on a single change?   +5 pts
• Have you linked the GitHub issue(s) that this PR addresses?   +5 pts
• Have you documented your changes clearly and comprehensively?   +5 pts
• Have you changed or added a feature?   -4 pts
  • Have you added/updated corresponding documentation?   +4 pts
  • Have you added/updated corresponding integration tests?   +5 pts
• Have you changed the behavior of AutoGPT?   -5 pts
  • Have you also run agbenchmark to verify that these changes do not regress performance?   +10 pts

[netlify]

✅ Deploy Preview for auto-gpt-docs canceled.

Name	Link
🔨 Latest commit	bac353e
🔍 Latest deploy log	https://app.netlify.com/sites/auto-gpt-docs/deploys/6697cbb90b0c8a00086c84e1

[github-actions]

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

[Swiftyos]

@CodiumAI-Agent /review

[CodiumAI-Agent]

PR Review

⏱️ Estimated effort to review [1-5]	4, due to the complexity and breadth of the changes introduced, including new model provider integrations, extensive modifications to configuration and provider logic, and the addition of new scripts and documentation. The PR touches multiple core components and introduces a new LLM provider, which requires careful review to ensure compatibility and correctness.
🧪 Relevant tests	No
🔍 Possible issues	Possible Bug: The method check_model_llamafile in configurator.py uses api_credentials.api_base.get_secret_value() which might expose sensitive information in error messages. This could lead to security risks if the error messages are logged or displayed in an environment where unauthorized users can view them.
	Possible Bug: In LlamafileProvider, the method _create_chat_completion hard-codes the seed for reproducibility, which might not be desirable in all use cases and could limit the functionality of the model in generating diverse responses.
🔒 Security concerns	Sensitive information exposure: The method check_model_llamafile potentially exposes sensitive API base URLs in exception messages, which could be a security risk if these messages are logged or improperly handled.

Code feedback:

---

relevant file	autogpts/autogpt/autogpt/app/configurator.py
suggestion	Consider removing or masking sensitive information such as api_base from error messages in check_model_llamafile to prevent potential leakage of sensitive data. [important]
relevant line	raise ValueError(f"llamafile server at {api_credentials.api_base.get_secret_value()} does not have access to {model_name}. Please configure {model_type} to use one of {available_model_ids} or use a different llamafile.")

---

relevant file	autogpts/autogpt/autogpt/core/resource/model_providers/llamafile.py
suggestion	Remove the hard-coded seed in _create_chat_completion or make it configurable via method parameters or configuration settings to allow for more dynamic behavior. [important]
relevant line	kwargs["seed"] = 0

---

---

✨ Review tool usage guide:

---

Overview:
The review tool scans the PR code changes, and generates a PR review which includes several types of feedbacks, such as possible PR issues, security threats and relevant test in the PR. More feedbacks can be added by configuring the tool.

The tool can be triggered automatically every time a new PR is opened, or can be invoked manually by commenting on any PR.

• When commenting, to edit configurations related to the review tool (pr_reviewer section), use the following template:

/review --pr_reviewer.some_config1=... --pr_reviewer.some_config2=...

• With a configuration file, use the following template:

[pr_reviewer]
some_config1=...
some_config2=...

See the review usage page for a comprehensive guide on using this tool.

[Pwuts]

@k8si any chance you could enable maintainer write access on this PR?

[k8si]

@Pwuts it doesn't look like I have the ability to do that. I added you as a maintainer to the forked project, is that sufficient or do others need write access?

Alternatively, you could branch off my branch and I can just accept the changes via PR?

[github-actions]

Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly.

[github-actions]

Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly.

[codecov]

Codecov Report

Attention: Patch coverage is 35.76159% with 97 lines in your changes missing coverage. Please review.

Project coverage is 53.81%. Comparing base (bffb92b) to head (bac353e).
Report is 30 commits behind head on master.

Files	Patch %	Lines
forge/forge/llm/providers/llamafile/llamafile.py	35.60%	85 Missing ⚠️
forge/forge/llm/providers/multi.py	25.00%	12 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #7091      +/-   ##
==========================================
- Coverage   54.21%   53.81%   -0.41%     
==========================================
  Files         122      124       +2     
  Lines        6875     7021     +146     
  Branches      881      909      +28     
==========================================
+ Hits         3727     3778      +51     
- Misses       3015     3110      +95     
  Partials      133      133              

Flag	Coverage Δ
Linux	53.56% <35.76%> (-0.40%)	⬇️
Windows	50.41% <35.76%> (-0.34%)	⬇️
autogpt-agent	33.99% <ø> (-0.03%)	⬇️
forge	58.04% <35.76%> (-0.60%)	⬇️
macOS	52.88% <35.76%> (-0.39%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[kcze]

That's 7? 🤔

[Pwuts]

@k8si can you clarify?

[ntindle]

this "works" but the model isn't great. Can we constrain the output schema to our models like is an option in the llamafile UI?

[Pwuts]

this "works" but the model isn't great. Can we constrain the output schema to our models like is an option in the llamafile UI?

I think we could implement something like that by allowing to pass a model as the completion_parser. Sounds like a follow-up PR though.

[github-actions]

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

[github-actions]

Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly.