Add Support for Memcached as a LLM Model Cache

[efriis]

Discussed in #27035

^{Originally posted by prokopchukdim October 1, 2024}

Checked

• I searched existing ideas and did not find a similar one
• I added a very descriptive title
• I've clearly described the feature request and motivation for it

Feature request

We would like to add support for Memcached as a usable LLM model cache. There are two main pure-Python memcached client implementations in Python: pymemcache and python-memcached.

We would primarily like to add support for pymemcache given that it is the most actively maintained, but it may be possible to support both clients under one newly added cache class since both are used.

Motivation

Many of the model caches supported natively are full on DBs. While Redis is supported as an option for distributed in-memory storage, many teams and companies rely on Memcached as a distributed in-memory cache. By adding Memcached support, we hope to make the model caching feature more useful to more teams using Langchain.

Example Usage

from langchain.globals import set_llm_cache
from langchain_openai import OpenAI

from langchain_community.cache import MemcachedCache
from pymemcache.client.base import Client

llm = OpenAI(model="gpt-3.5-turbo-instruct", n=2, best_of=2)
set_llm_cache(MemcachedCache(Client('localhost')))

# The first time, it is not yet in cache, so it should take longer
llm.invoke("Which city is the most crowded city in the USA?")

# The second time it is, so it goes faster
llm.invoke("Which city is the most crowded city in the USA?")

Proposal (If applicable)

We intend to add a new MemcachedCache implementation in libs/community/langchain_community/cache.py to support the pymemcache client.

If there is interest in also supporting the python-memcached client, or others, we can explore creating a unified implementation class since all clients should generally adhere to the memcached text protocol.

We intend to submit a pull request some time in October, and no later than mid-November.

[prokopchukdim]

Thank you for creating an issue for us! We should have a PR created in a week or two. Could this issue be assigned to me?

[prokopchukdim]

@efriis We just created a PR for this integration, would appreciate if you could take a look!