Resolving discrepancy when image model is different than base model

[simistern]

Describe the bug

During experimentation, I have found that it was not clear when the image model was not in sync with the base character model. For example, grok image generation API is not publicly available yet, so when asking my character to generate an image, it defaulted to openAI but that code was not set. It is not clear to developers that this is possible (the image gen using a different fallback than the base LLM). In the case where the image model is provided but does not work, it will assume the underlying access token is openAI, and will use groks token. I think this will extend to all other models where the image model API is not aligned with this core/generation file.

I have added some logging and a bit of logic to improve this and will take ownership of integrating grok image gen when it becomes publicly available, I am following along development in the xAI discord.

Currently seeking repo permissions to submit the PR for the enhancement to packages/core/src/generation and to add grok image gen capability when it becomes available.

To Reproduce

Set character model to grok and ask it to generate an image, user will receive error message pointing to openAI env keys not being set.

Expected behavior

User is informed that the image does not exist or will not match the base model, and that it will default to OPENAI

Screenshots

Additional context

[github-actions]

Hello @simistern! Welcome to the ai16z community. Thank you for opening your first issue; we appreciate your contribution. You are now a ai16z contributor!

[AIFlowML]

Describe the bug

During experimentation, I have found that it was not clear when the image model was not in sync with the base character model. For example, grok image generation API is not publicly available yet, so when asking my character to generate an image, it defaulted to openAI but that code was not set. It is not clear to developers that this is possible (the image gen using a different fallback than the base LLM). In the case where the image model is provided but does not work, it will assume the underlying access token is openAI, and will use groks token. I think this will extend to all other models where the image model API is not aligned with this core/generation file.

I have added some logging and a bit of logic to improve this and will take ownership of integrating grok image gen when it becomes publicly available, I am following along development in the xAI discord.

Currently seeking repo permissions to submit the PR for the enhancement to packages/core/src/generation and to add grok image gen capability when it becomes available.

To Reproduce

Set character model to grok and ask it to generate an image, user will receive error message pointing to openAI env keys not being set.

Expected behavior

User is informed that the image does not exist or will not match the base model, and that it will default to OPENAI

Screenshots

Additional context

It has been like this since a bit.
the image related models are called into the pluing too as OAI.
In the next release we wil for sure solve this.

[tcm390]

Hi @simistern, are you working on resolving this issue?

[simistern]

Hi @tcm390, I have resolved the issue on a local branch but I think I need permissions to submit the PR? Is there a process for being allowed to submit PR's?

[simistern]

Describe the bug
During experimentation, I have found that it was not clear when the image model was not in sync with the base character model. For example, grok image generation API is not publicly available yet, so when asking my character to generate an image, it defaulted to openAI but that code was not set. It is not clear to developers that this is possible (the image gen using a different fallback than the base LLM). In the case where the image model is provided but does not work, it will assume the underlying access token is openAI, and will use groks token. I think this will extend to all other models where the image model API is not aligned with this core/generation file.
I have added some logging and a bit of logic to improve this and will take ownership of integrating grok image gen when it becomes publicly available, I am following along development in the xAI discord.
Currently seeking repo permissions to submit the PR for the enhancement to packages/core/src/generation and to add grok image gen capability when it becomes available.
To Reproduce
Set character model to grok and ask it to generate an image, user will receive error message pointing to openAI env keys not being set.
Expected behavior
User is informed that the image does not exist or will not match the base model, and that it will default to OPENAI
Screenshots
Additional context

It has been like this since a bit. the image related models are called into the pluing too as OAI. In the next release we wil for sure solve this.

@AIFlowML I have fixed this locally, happy to submit the PR and become a contributor :)

[AIFlowML]

The user solved locally the problem and is ready to PR
Yes @simistern send over the PR we will review it.