-
Notifications
You must be signed in to change notification settings - Fork 3.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: add image text model provider separation and fal.ai integration #650
Merged
lalalune
merged 13 commits into
elizaOS:main
from
yoniebans:feat/add_image_text_model_provider_seperation_and_falai
Nov 29, 2024
Merged
feat: add image text model provider separation and fal.ai integration #650
lalalune
merged 13 commits into
elizaOS:main
from
yoniebans:feat/add_image_text_model_provider_seperation_and_falai
Nov 29, 2024
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…Provider allowing for use of different models between textGeneration and imageGeneration. If imageModelProvider is not set, it defaults to modelProvider and as such, functionality should remain the same. Also added another Model and ModelProviderName for fal.ai
…erateText and generateImage, added a new property for the imageModelProvider on the runtime class. This property is set to the imageModelProvider on character if present, otherwise it defaults to the model provider hence, the same model will be used for bother generateText and generateImage
… FAL_API_KEY not that it can be used for generateImage
…ion and leverage the new distinction between imageModelProvider and modelProvider. Have changed the checks from runtime.character.modelProvider to runtime.modelProvider as it should be set upon runtime intialisation for character
yoniebans
changed the title
feat: add image text model provider seperation and fal.ai integration
feat: add image text model provider separation and fal.ai integration
Nov 28, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Relates to:
Risks
Low - Adds optional fal.ai integration for image generation. Existing functionality remains unchanged when new optional imageModelProvider is not set.
Background
What does this PR do?
FAL_API_KEY
andFAL_AI_LORA_PATH
environment variablesWhat kind of change is this?
Features (non-breaking change which adds functionality)
Documentation changes needed?
My changes require a change to the project documentation to document:
Testing
Where should a reviewer start?
agent/src/index.ts
- Updated to include image generation plugin should one of the valid image generation model api keys be included.packages/core/src/generation.ts
- Review fal.ai integration. Changed to look at model provider on runtime instance.if modelProvider = imageModelProvider
, use token otherwise use one of the image env vars set.packages/core/src/models.ts
- Added new entry for FAL modelpackages/core/src/types.ts
- Check newimageModelProvider
type additionspackages/core/src/runtime.ts
- ReviewimageModelProvider
implementation. Updated to setimageModelProvider
if one is provided in character config otherwise default tomodelProvider
.Detailed testing steps
fal ai api testing:
FAL_API_KEY
to .envimageModelProvider
: "falai"FAL_AI_LORA_PATH
same
modelProvider
toimageModelProvider
tests:OPENAI_API_KEY
modelProvider
to "openai" in character configdifferent
modelProvider
toimageModelProvider
tests:OPENAI_API_KEY
&FAL_API_KEY
modelProvider
to "openai" andimageModelProvider
to "falai" in character configdifferent
modelProvider
toimageModelProvider
tests:OPENAI_API_KEY
&HEURIST_API_KEY
modelProvider
to "openai" andimageModelProvider
to "heurist" in character configDiscord username
yoniebans