feat: add image features to react chat client #1481
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Relates to: Images in the react chat client
Risks
Low - Changes are isolated to image handling functionality and don't affect core messaging or authentication systems.
Background
I wanted to be able to upload images in the built in React client, and have the agent be able to respond with image attachments using various plugins.
What does this PR do?
This PR implements image handling capabilities by making some additions in a couple packages.
/:agentId/message
POST endpoint.describeImage
Action in the plugin to take advantage of theImageDescriptionService
in the same plugin.recognizeWithOpenAI
andrequestOpenAI
to accept local files, similar to how the first frame of .gif files is converted to a buffer and sent over the APIWhat kind of change is this?
Features (non-breaking change which adds functionality)
Documentation changes needed?
NA
Testing
Where should a reviewer start?
packages/plugin-node/src/services/image.ts
- Core image processing serviceDetailed testing steps
Test image upload:
describeImage
actionTest generate image: