Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add image features to react chat client #1481

Merged
merged 5 commits into from
Dec 27, 2024

Conversation

0xPBIT
Copy link
Contributor

@0xPBIT 0xPBIT commented Dec 26, 2024

Relates to: Images in the react chat client

Risks

Low - Changes are isolated to image handling functionality and don't affect core messaging or authentication systems.

Background

I wanted to be able to upload images in the built in React client, and have the agent be able to respond with image attachments using various plugins.

What does this PR do?

This PR implements image handling capabilities by making some additions in a couple packages.

  1. Client App:
  • Added image display support in the chat interface
  • Implemented image upload functionality
  • Enhanced UI to handle image messages
  1. Client Direct:
  • Implemented image attachment handling in the /:agentId/message POST endpoint.
  • Added static file hosting for the direct client so that generated images and uploaded images can be displayed in the client.
  1. Plugin Node:
  • Added a describeImage Action in the plugin to take advantage of the ImageDescriptionService in the same plugin.
  • Modified recognizeWithOpenAI and requestOpenAI to accept local files, similar to how the first frame of .gif files is converted to a buffer and sent over the API

What kind of change is this?

Features (non-breaking change which adds functionality)

Documentation changes needed?

NA

Testing

Where should a reviewer start?

  1. packages/plugin-node/src/services/image.ts - Core image processing service
  2. Client app image upload functionality

Detailed testing steps

  1. Test image upload:

    • Upload various image formats (JPG, PNG, GIF)
    • Verify proper display in chat
    • Confirm image descriptions are generated using built in describeImage action
  2. Test generate image:

    • Request the agent generate an image using the built in image generator plugin
    • Verify proper display in chat
Screenshot 2024-12-26 at 3 39 18 PM

@0xPBIT 0xPBIT force-pushed the images-in-chat-client branch from bacaa79 to 604f460 Compare December 26, 2024 22:02
@0xPBIT 0xPBIT force-pushed the images-in-chat-client branch from 604f460 to b274b5c Compare December 26, 2024 22:06
@shakkernerd
Copy link
Member

Hi @0xPBIT kindly resolve conflicts.

@0xPBIT
Copy link
Contributor Author

0xPBIT commented Dec 27, 2024

thank you @monilpat

shakkernerd
shakkernerd previously approved these changes Dec 27, 2024
Copy link
Member

@shakkernerd shakkernerd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is nice!
Looks good to me!

@shakkernerd shakkernerd merged commit 8d7f67d into elizaOS:develop Dec 27, 2024
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants