Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add image text model provider separation and fal.ai integration #650

Conversation

yoniebans
Copy link
Contributor

@yoniebans yoniebans commented Nov 28, 2024

Relates to:

Risks

Low - Adds optional fal.ai integration for image generation. Existing functionality remains unchanged when new optional imageModelProvider is not set.

Background

What does this PR do?

  • Adds fal.ai integration for image generation
  • Introduces separate optional image model provider configuration
  • Adds FAL_API_KEY and FAL_AI_LORA_PATH environment variables
  • Implements fal.ai client setup and image generation logic

What kind of change is this?

Features (non-breaking change which adds functionality)

Documentation changes needed?

My changes require a change to the project documentation to document:

  • New environment variables
  • How to configure fal.ai image generation
  • Using separate model providers for text and image generation

Testing

Where should a reviewer start?

  1. agent/src/index.ts - Updated to include image generation plugin should one of the valid image generation model api keys be included.
  2. packages/core/src/generation.ts - Review fal.ai integration. Changed to look at model provider on runtime instance. if modelProvider = imageModelProvider, use token otherwise use one of the image env vars set.
  3. packages/core/src/models.ts - Added new entry for FAL model
  4. packages/core/src/types.ts - Check new imageModelProvider type additions
  5. packages/core/src/runtime.ts - Review imageModelProvider implementation. Updated to set imageModelProvider if one is provided in character config otherwise default to modelProvider.

Detailed testing steps

fal ai api testing:

  1. Add FAL_API_KEY to .env
  2. Configure character with imageModelProvider: "falai"
  3. Test image generation with default settings
  4. Test with custom FAL_AI_LORA_PATH
  5. Verify existing providers still work when "falai" is not used
  6. Test switching between different image providers

same modelProvider to imageModelProvider tests:

  1. Set OPENAI_API_KEY
  2. Set modelProvider to "openai" in character config
  3. Both text generation and image generation goes through openai

different modelProvider to imageModelProvider tests:

  1. Set OPENAI_API_KEY & FAL_API_KEY
  2. Set modelProvider to "openai" and imageModelProvider to "falai" in character config
  3. text generation goes through openai and image generation goes through falai

different modelProvider to imageModelProvider tests:

  1. Set OPENAI_API_KEY & HEURIST_API_KEY
  2. Set modelProvider to "openai" and imageModelProvider to "heurist" in character config
  3. text generation goes through openai and image generation goes through heurist

Discord username

yoniebans

…Provider allowing for use of different models between textGeneration and imageGeneration. If imageModelProvider is not set, it defaults to modelProvider and as such, functionality should remain the same. Also added another Model and ModelProviderName for fal.ai
…erateText and generateImage, added a new property for the imageModelProvider on the runtime class. This property is set to the imageModelProvider on character if present, otherwise it defaults to the model provider hence, the same model will be used for bother generateText and generateImage
… FAL_API_KEY not that it can be used for generateImage
…ion and leverage the new distinction between imageModelProvider and modelProvider. Have changed the checks from runtime.character.modelProvider to runtime.modelProvider as it should be set upon runtime intialisation for character
@yoniebans yoniebans changed the title feat: add image text model provider seperation and fal.ai integration feat: add image text model provider separation and fal.ai integration Nov 28, 2024
@lalalune lalalune merged commit b1c0f56 into elizaOS:main Nov 29, 2024
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants