core[patch]: extract input variables for path and detail keys in order to format an ImagePromptTemplate

[thdesc]

• Description: Add support for path and detail keys in ImagePromptTemplate. Previously, only variables associated with the url key were considered. This PR allows for the inclusion of a local image path and a detail parameter as input to the format method.
• Issues:
  • fixes How to use ImagePromptTemplate #20820
  • related to Dinamically format HumanMessage list of dictionaries for multimodal LLM #22024
• Dependencies: None
• Twitter handle: @DeschampsTho5

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment

Name	Status	Preview	Comments	Updated (UTC)
langchain	⬜️ Ignored (Inspect)	Visit Preview		Jul 3, 2024 6:43pm

[thdesc]

Hi @eyurtsev do you have some time to review this PR soon ? I'm planning to take some time off, so I won't be able to iterate on your reviews afterward. Thanks!

[thdesc]

Hi @baskaryan @eyurtsev @efriis anyone of you have some time to review this PR please ? This should allow users to pass a path (local path to an image file) and a detail parameter (to control how the LLM processes the image) when formatting an HumanMessagePromptTemplate with a image_url.

Here is an example of how it would work:

from langchain_core.prompts import HumanMessagePromptTemplate, ChatPromptTemplate
from langchain_core.messages import HumanMessage

image_path = 'path_to_your_image.jpg'
detail_parameter = 'high'

chat_prompt_template = ChatPromptTemplate.from_messages(
    messages=[
        HumanMessage(content='Describe the following image.'),
        HumanMessagePromptTemplate.from_template(
            [{'image_url': {'path': {image_path}, 'detail': {detail_parameter}}]
        )
    ]
)

prompt = chat_prompt_template.format(image_path=image_path, detail_parameter=detail_parameter)

As you can see in this discussion #20820 there are several questions about how to use the ImagePromptTemplate with a HumanMessagePromptTemplate and I think this PR could help.

Thank you !

[thdesc]

Currently to upload a base 64 encoded image, users have to do something like this:

import base64

from langchain_core.prompts import HumanMessagePromptTemplate, ChatPromptTemplate
from langchain_core.messages import HumanMessage

def encode_image(image_path):
  with open(image_path, "rb") as image_file:
    return base64.b64encode(image_file.read()).decode('utf-8')

# Path to your image
image_path = "path_to_your_image.jpg"

# Getting the base64 string
base64_image = encode_image(image_path)

chat_prompt_template = ChatPromptTemplate.from_messages(
    messages=[
        HumanMessage(content='Describe the following image.'),
        HumanMessagePromptTemplate.from_template(
            [{'image_url': {'url': 'data:image/jpeg;base64,{base64_image}', 'detail': 'high'}]  # the detail parameter has to be hard-coded and can't be formatted at runtime.
        )
    ]
)

prompt = chat_prompt_template.format(base64_image=base64_image)

[eyurtsev]

On mobile right now so won't be able to review properly but from the PR description the API doesn't look correct

chat_prompt_template = ChatPromptTemplate.from_messages(
    messages=[
        HumanMessage(content='Describe the following image.'),
        HumanMessagePromptTemplate.from_template(
            [{'image_url': {'path': {image_path}, 'detail': {detail_parameter}}]
        )
    ]
)

prompt = chat_prompt_template.format(image_path=image_path, detail_parameter=detail_parameter)

Image path has already been set when defining the human prompt template. It doesn't appear that format does anything?

[thdesc]

@eyurtsev Oh yes you are 100% right sorry for that. It is a mistake in the comment only not in the PR I think. I should have wrote this:

from langchain_core.prompts import HumanMessagePromptTemplate, ChatPromptTemplate
from langchain_core.messages import HumanMessage

image_path = 'path_to_your_image.jpg'
detail_parameter = 'high'

chat_prompt_template = ChatPromptTemplate.from_messages(
    messages=[
        HumanMessage(content='Describe the following image.'),
        HumanMessagePromptTemplate.from_template(
            [{'image_url': {'path': '{image_path}', 'detail': '{detail_parameter}'}]
        )
    ]
)

prompt = chat_prompt_template.format(image_path=image_path, detail_parameter=detail_parameter)

[thdesc]

Hi @eyurtsev, did you have some time to review this PR please ?

[eyurtsev]

@thdesc thanks for updating the PR! Merging now :)

[thdesc]

@eyurtsev Thanks for the merge! Sorry I wasn't available last week and didn't have my computer to fix the broken test. I want to create another PR related to the ImagePromptTemplate. I will do it soon. Thanks again!