Image Description Service, unsupported image

[denizekiz]

Describe the bug

Getting unsupported image error from openai api. Despite image being a jpg.

To Reproduce
"develop" branch
pnpm start
Run twitter client
tag your agent with a photo on twitter/x

Expected behavior

⛔ ERRORS
OpenAI API error:
400

{
"error": {
"message": "You uploaded an unsupported image. Please make sure your image has of one the following formats: ['png', 'jpeg', 'gif', 'webp'].",
"type": "invalid_request_error",
"param": null,
"code": "invalid_image_format"
}
}

Screenshots

Additional context

[AIFlowML]

What is the version of ElizaOS you running ? Any other detail about the image ?

[denizekiz]

@AIFlowML You can reproduce by tagging your agent on twitter, and add any image into tweet to trigger. I use develop branch.

[tcm390]

@denizekiz is the issue resolved by #1696?

[denizekiz]

@tcm390 it only resolved when I call directly the service.

It still throw error if called with process actions in interaction.ts. Process action loop sends the url of the tweet to openai instead of url of image

[tcm390]

Thanks for the clarification I will check it 🙏

[tcm390]

@tcm390 it only resolved when I call directly the service.

It still throw error if called with process actions in interaction.ts. Process action loop sends the url of the tweet to openai instead of url of image

I think it's fixed? Can you confirm if this issue is fixed by this PR? #1671

[denizekiz]

Unfortunately no. In interaction.ts message.url is tweet url. So it passed to ImageDescriptionService when agent decide to take "describeimage" action. Need to pass photo url so that it can handle.

    const message = {
            id: tweetId,
            agentId: this.runtime.agentId,
            content: {
                text: tweet.text,
                url: tweet.permanentUrl,
                inReplyTo: tweet.inReplyToStatusId
                    ? stringToUuid(
                          tweet.inReplyToStatusId +
                              "-" +
                              this.runtime.agentId
                      )
                    : undefined,
            },
            userId: userIdUUID,
            roomId,
            createdAt: tweet.timestamp * 1000,
        };
        this.client.saveRequestMessage(message, state);
    }
    
    
     await this.runtime.processActions(
                message,
                responseMessages,
                state,
                callback
            );

[tcm390]

I see. Okay, I will look into it. Thanks for the clarification!

[AIFlowML]

Working on it.
Both 1694 and 1668

[AIFlowML]

ImageDescriptionService is the test now is working.
Im on the twiter client part now.

[AIFlowML]

I add some more error handling also in the client it seem all goods now.
I wrap up and PR

[denizekiz]

@AIFlowML the message in process action only takes tweet's url, not image url, which creates problem once its called with processaction logic called from runtime in twitter client.

Once agent selects describe image and Runtime.processaction tries calling imagedescription service in twitter client's interactions.ts, it gets tweet url instead of tweet photo urls.

[denizekiz]

Seems like llm infer wrong url

https://github.com/elizaOS/eliza/blob/main/packages/plugin-node/src/actions/describe-image.ts

[AIFlowML]

@AIFlowML the message in process action only takes tweet's url, not image url, which creates problem once its called with processaction logic called from runtime in twitter client.

Once agent selects describe image and Runtime.processaction tries calling imagedescription service in twitter client's interactions.ts, it gets tweet url instead of tweet photo urls.

I submitted a PR yesterday and tested the imagedescription + fixed the twitter that was using it it also now have logging and error handling so in case we know what is wrong.

[AIFlowML]

Probably fixed by #1756
I wil wait the merge to close this.