feat: refactor `CoreMemory` to support generalized memory fields and memory editing functions #1479

sarahwooders · 2024-06-26T00:51:46Z

Please describe the purpose of this pull request.
This PR refactors the code to allow for customizable memory modules and deprecates usage of presets for agent creation. I also implemented functionality to match the LocalClient and RESTClient to make testing easier.

Generalized Memory Class

This PR introduces to create custom memory classes that have custom fields (must be either a str of List[str]) and custom memory editing functions, inspired by work in #895.

Sections of core memory are defined by a MemoryModule class (corresponds to sections of the in-context memory)

class MemoryModule(BaseModel):
    """Base class for memory modules"""

    description: Optional[str] = None
    limit: int = 2000
    value: Optional[Union[List[str], str]] = None

The default memory class is implemented on top of a base BaseMemory class:

class ChatMemory(BaseMemory):

    def __init__(self, persona: str, human: str, limit: int = 2000):
        self.memory = {
            "persona": MemoryModule(name="persona", value=persona, limit=limit),
            "human": MemoryModule(name="human", value=human, limit=limit),
        }

    def core_memory_append(self, name: str, content: str) -> Optional[str]:
        """
        Append to the contents of core memory.

        Args:
            name (str): Section of the memory to be edited (persona or human).
            content (str): Content to write to the memory. All unicode (including emojis) are supported.

        Returns:
            Optional[str]: None is always returned as this function does not produce a response.
        """
        self.memory[name].value += "\n" + content
        return None

    def core_memory_replace(self, name: str, old_content: str, new_content: str) -> Optional[str]:
        """
        Replace the contents of core memory. To delete memories, use an empty string for new_content.

        Args:
            name (str): Section of the memory to be edited (persona or human).
            old_content (str): String to replace. Must be an exact match.
            new_content (str): Content to write to the memory. All unicode (including emojis) are supported.

        Returns:
            Optional[str]: None is always returned as this function does not produce a response.
        """
        self.memory[name].value.replace(old_content, new_content)
        return None

Improve Agent Creation

Agent creation is modified to take in a memory class instead of human/persona/preset:

    def create_agent(
        self,
        name: Optional[str] = None,
        # model configs
        embedding_config: Optional[EmbeddingConfig] = None,
        llm_config: Optional[LLMConfig] = None,
        # memory
        memory: BaseMemory = ChatMemory(human=get_human_text(DEFAULT_HUMAN), persona=get_human_text(DEFAULT_PERSONA)),
        # tools
        tools: Optional[List[str]] = None,
        include_base_tools: Optional[bool] = True,
        # metadata
        metadata: Optional[Dict] = {"human:": DEFAULT_HUMAN, "persona": DEFAULT_PERSONA},
    ) -> AgentState:

Deprecated

Deprecation of human, persona and preset fields from agent creation
Deprecation of creating agents from preset

How to test
poetry run pytest -s tests/test_memory.py

Have you tested this PR?
Yes

Related issues or PRs
#895

Maximilian-Winter · 2024-06-29T10:07:25Z

Hi, I just wanted to show you @sarahwooders what I did in my MemGPT version to make the system prompt generation customizable. Similar to your MemoryModule, I introduced the SystemPromptModules, which are basically sections added to the system prompt.

memory_prompt = """1. Core Memory - Stores essential context about the user, your persona and your current scratchpad, it is divided into a user section, a persona section and your scratchpad section. You can use the scratchpad to plan your next actions. You can edit the core memory by calling the functions: 'core_memory_append', 'core_memory_remove' and 'core_memory_replace'.

2. Archival Memory - Archive to store and retrieve general information and events about the user and your interactions with it. Can be used by calling the functions: 'archival_memory_search' and 'archival_memory_insert'.

3. Conversation History - Since you are only seeing the latest conversation history, you can search the rest of the conversation history. Search it by using: 'conversation_search' and 'conversation_search_date'.

Always remember that the user can't see your memory or your interactions with it!"""

memory_intro_section = SystemPromptModule(section_name="memory_intro",
                                          prefix="To support you in your task as a AI assistant and to help you remembering things, you have access to 3 different types of memory.",
                                          position=SystemPromptModulePosition.after_system_instructions)
memory_intro_section.set_content(memory_prompt)

You can modify the content at runtime.

sarahwooders · 2024-07-01T16:34:52Z

Hi, I just wanted to show you @sarahwooders what I did in my MemGPT version to make the system prompt generation customizable. Similar to your MemoryModule, I introduced the SystemPromptModules, which are basically sections added to the system prompt.

memory_prompt = """1. Core Memory - Stores essential context about the user, your persona and your current scratchpad, it is divided into a user section, a persona section and your scratchpad section. You can use the scratchpad to plan your next actions. You can edit the core memory by calling the functions: 'core_memory_append', 'core_memory_remove' and 'core_memory_replace'.

2. Archival Memory - Archive to store and retrieve general information and events about the user and your interactions with it. Can be used by calling the functions: 'archival_memory_search' and 'archival_memory_insert'.

3. Conversation History - Since you are only seeing the latest conversation history, you can search the rest of the conversation history. Search it by using: 'conversation_search' and 'conversation_search_date'.

Always remember that the user can't see your memory or your interactions with it!"""

memory_intro_section = SystemPromptModule(section_name="memory_intro",
                                          prefix="To support you in your task as a AI assistant and to help you remembering things, you have access to 3 different types of memory.",
                                          position=SystemPromptModulePosition.after_system_instructions)
memory_intro_section.set_content(memory_prompt)

You can modify the content at runtime.

Thanks for sharing this! We're planning to also modularize the system prompt sections of the context as well, but will do it in another PR to try to control the scope of this one.

memgpt/agent.py

cpacker · 2024-07-01T18:14:40Z

memgpt/agent.py

@@ -912,146 +847,111 @@ def heartbeat_is_paused(self):

    def rebuild_memory(self):
        """Rebuilds the system message with the latest memory object"""
+        print("rebuild memory")


also should remove these prints

memgpt/agent.py

cpacker · 2024-07-01T18:16:08Z

memgpt/agent.py

-        self.update_state()
-        printd(msg)
-        return msg
+        # TODO: refactor


probably should raise NotImplementedError instead?

or raise a Warning?

import warnings warnings.warn("...")

I did a NotImplementedError

cpacker

Need to strip stray prints, will test in CLI now

cpacker · 2024-07-01T18:19:37Z

Also should we be adding a migration script for this pre-release?

VALIDATE None {'description': None, 'limit': 2000}
VALIDATE First name: Chad {'description': None, 'limit': 2000}
VALIDATE None {'description': None, 'limit': 2000}
VALIDATE First name: Chad {'description': None, 'limit': 2000}
Traceback (most recent call last):
  File "/Users/loaner/Library/Caches/pypoetry/virtualenvs/pymemgpt-JSsUGnlY-py3.10/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context
    self.dialect.do_execute(
  File "/Users/loaner/Library/Caches/pypoetry/virtualenvs/pymemgpt-JSsUGnlY-py3.10/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 924, in do_execute
    cursor.execute(statement, parameters)
sqlite3.OperationalError: no such column: agents._metadata

  File "/Users/loaner/dev/MemGPT-fresh/memgpt/metadata.py", line 613, in list_agents
    results = session.query(AgentModel).filter(AgentModel.user_id == user_id).all()
...
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such column: agents._metadata
[SQL: SELECT agents.id AS agents_id, agents.user_id AS agents_user_id, agents.name AS agents_name, agents.system AS agents_system, agents.created_at AS agents_created_at, agents.llm_config AS agents_llm_config, agents.embedding_config AS agents_embedding_config, agents.state AS agents_state, agents._metadata AS agents__metadata, agents.tools AS agents_tools 
FROM agents 
WHERE agents.user_id = ?]

cpacker

LGTM

cpacker · 2024-07-01T18:51:06Z

Thank you @Maximilian-Winter ! <3

norton120

I'd like to pull this into #1460 once it's merged and update the datamodel to match - I think this will really help clear up the configs too

norton120 · 2024-07-02T20:03:10Z

memgpt/memory.py


+class MemoryModule(BaseModel):


Looking at this section here I'm not entirely clear on what a MemoryModule is vs a BaseMemory, and by association, I'm not clear on what the attributes represent (so it's hard to follow along).

More unique names, descriptive class docstrings, and Field annotations for attributes with descriptions would help.

For example, is the limit attribute on MemoryModule the number of characters in a Memory, or the number of Memory objects, or something else?

value is another one - it almost looks like that's another subschema here. I'm not sure what values are compared to memories.

The limit is how many characters can be in the string representation of value, since the string representation of value is placed into the context window for the LLM.

We are actually planning to eventually store the MemoryModule (or whatever its renamed to) as a DB row that is read from at inference time (when the context is "compiled"). This would allow multiple agents to share a given row (e.g. where value represents organizational memory across agents). So maybe another potential name for the class could be StateVariable or ContextVariable.

norton120 · 2024-07-02T20:04:31Z

memgpt/memory.py


+class MemoryModule(BaseModel):


Module is a reserved word in Python - modules are the things you import from files. MemoryModule also semantically very close to Memory which makes it harder to code and easier to mess up.

The general rule is if you find a whole bunch of things having the same or similar name ("Delivery", "ShipperDelivery", "VendorDelivery") don't allow any of them to use the overused words. Then the names you are left with are distinct and you avoid confusion i.e. ("CustomerReceipt", "CarrierShipment", "VendorReceipt").

I just used the term "Thought" below because it was what made sense when typing (memories are made up of thoughts) - I'm not suggesting that be the name, but I would recommend getting a nice wide Levenshtein distance between the names so it's easy to remember which is which when coding (and thinking)

norton120 · 2024-07-02T20:05:53Z

memgpt/memory.py

-        self.human = human
-        self.persona_char_limit = persona_char_limit
-        self.human_char_limit = human_char_limit
+    @validator("value", always=True)


If this is supposed to be defaulting to the class limit it should only be set there.

norton120 · 2024-07-02T20:07:34Z

memgpt/memory.py

-        # affects the error message the AI will see on overflow inserts
-        self.archival_memory_exists = archival_memory_exists
+            # Check if the value exceeds the limit
+            if isinstance(v, str):


are these length values set in code downstream? or passed as instance values? if they are code this can just be a field prop (as a value, then I agree this is probably the best/most readable option)

what do you mean by the length values? Both the value and potentially the limit (though we don't support this now) could be modified for an existing agent.

Sorry, I was thinking ahead of my words with code - the "limit" values, not "length" values 🤦🏻

basically, what I was trying to figure out by looking at the 2 classes is if one is a shell for the other and the limit is a default, or if the limit is the total that all the values must add up to. It's still not super clear (classic "naming things is the hardest part of programming" thing).

Going through this PR review has been helpful - I think I get most of the relationships now, the values are intended to be the actual string content of a given section/partial of core memory, and it can be a single string or many strings that get concatenated into a single utterance within the context window.

It would be solid to add this context into the objects (using class docstrings and Field description attributes) - it doesn't need to be in this PR. That's a great "first issue" to take the knowledge from these discussions and instrument the code base.

norton120 · 2024-07-02T20:10:30Z

memgpt/memory.py

+            return ""
+
+
+class BaseMemory:


how come this isn't a pydantic model?

I guess it should be yeah... I forgot that they can also have class functions...

norton120 · 2024-07-02T20:16:18Z

memgpt/memory.py

+            obj.memory[key] = MemoryModule(**value)
+        return obj
+
+    def __str__(self) -> str:


It feels like there's a desired schema (it's implied) for memory objects - that would be a good thing to make another pydantic object and validate here.

norton120 · 2024-07-02T20:17:48Z

memgpt/memory.py

-            return self.edit_human(new_content)
-        else:
-            raise KeyError(f'No memory section named {field} (must be either "persona" or "human")')
+    def core_memory_append(self, name: str, content: str) -> Optional[str]:


How come core_memory_apend is in the child method and not the base method?

The docstrings are specific to the ChatMemory class (they mention human/persona) so would need to be overriden by customized memory.

norton120 · 2024-07-02T20:34:58Z

memgpt/memory.py

+
+    def __init__(self, persona: str, human: str, limit: int = 2000):
+        self.memory = {
+            "persona": MemoryModule(name="persona", value=persona, limit=limit),


Ahh ok, so is a MemoryModule the contents of a section within a BaseMemory?

If I can make a suggestion, the BaseMemory.memory structure is awkward as a dict of typed objects with duplicate props. It seems like it should be a list (since that also preserves order, in case you care about that).

Something like

>> ChatMemory.thoughts [<class 'MemoryModule' persona 145>, <class 'MemoryModule' human 492>, <class 'MemoryModule' company 941>]

There's a data normalization issue with the duplication here, you can have this:

>> chat_memory.memory.persona <class 'MemoryModule' human 1998> # This shouldn't be possible!

If it's about lookups, there's some easy metaprogramming we can do for that

def __getattr__(self, val): try: return next([m in self.thoughts if m.name == 'val']) except StopIteration as e: raise AttributeError(f'{self.__class__.__name__} has no attribute {val}') from e

which looks like this in practice

>> chat_memory.persona <class 'MemoryModule' persona 2401> >> chat_memory.three_laws <class 'MemoryModule' three_laws 100> >> chat_memory.hamburger AttributeError: ChatMemory has no attribute 'hamburger'

there would need to be a little extra to enforce only a single MemoryModule of a given type, unless you want to be able to have more than one persona or human? worth a thought

Yeah the MemoryModule basically represents a reserved section of the context (or at least allocation of a context up to the size limit) for placing the actual text value of the section of memory - we will also probably generalize this eventually to also include the system prompt (not just core memory), and store things like pre-defined human/persona values or system prompts under the same schema.

We also considered Variable, Snipped, and ContextSlice - Thoughts overlaps with inner-thoughts which is very different so I think that might be confusing.

^^^ bingo! if we can add this into the code even as a docstring, it'll help when the name is a little sticky.

So if a Memory is the entire context window (like the core memory computing paradigm),
are these slots or registers or MemoryPartials...

But I have held this PR up enough, and perfect is the enemy of gsd. I vote roll with MemoryModule until a more natural object name comes to mind, then migrate.

…memory editing functions (#1479) Co-authored-by: cpacker <[email protected]> Co-authored-by: Maximilian-Winter <[email protected]>

refactor memory initial commit

4afdec5

sarahwooders requested a review from cpacker June 26, 2024 00:51

sarahwooders added 2 commits June 25, 2024 20:39

update create agent

c494d54

add todo

7952793

sarahwooders mentioned this pull request Jun 26, 2024

feat: Dynamic template system with editable system prompt fields. #895

Closed

sarahwooders added 10 commits June 26, 2024 20:48

Merge branch 'main' into refactor-memory

8df27e9

Merge branch 'main' into refactor-memory

32a3801

add tests

907faf1

working agent creation for local

070b404

update create

e083162

working CLI

5c22be9

remove old function code

690c4fd

update get/set item to enable proper function linking

e8c84d4

fix bug in update

f4e8456

add memory tools

4146ddd

sarahwooders added 14 commits June 30, 2024 12:30

merge

909793f

passing for both local and rest client

6d6506a

cli fixes

f9b4d8b

update memory

7b637e1

implement local archival memory

58ac373

fix local messages

124158d

fix human/persona

680da48

fix human/persona

858b2d1

fix human/persona

ea9e273

fix tests

b48a1d9

fix tests

32a3e92

fix tests

5ba118f

fix server tests

fe24dad

fix server tests

12522a5

sarahwooders added 7 commits June 30, 2024 18:33

metadata

08be41f

fix cli

0343f5e

fix tools

22bc17d

fix test base functions

0b1d780

fix cli

6b8c8d9

fix storage

9debbea

fix docker tests

c554ed9

sarahwooders added 2 commits July 1, 2024 10:54

cleanup

6231c5c

more cleanup

2f30257

cpacker reviewed Jul 1, 2024

View reviewed changes

memgpt/agent.py Outdated Show resolved Hide resolved

cpacker reviewed Jul 1, 2024

View reviewed changes

memgpt/agent.py Outdated Show resolved Hide resolved

cpacker reviewed Jul 1, 2024

View reviewed changes

cpacker requested changes Jul 1, 2024

View reviewed changes

sarahwooders added 2 commits July 1, 2024 11:28

more cleanup

3160a70

more cleanup

d526403

sarahwooders requested a review from cpacker July 1, 2024 18:31

cpacker added 2 commits July 1, 2024 11:33

typo

67b0ed7

make cli update the server logger settings too

7fe4748

cpacker approved these changes Jul 1, 2024

View reviewed changes

cpacker merged commit c9f62f5 into main Jul 1, 2024
11 checks passed

cpacker deleted the refactor-memory branch July 1, 2024 18:51

norton120 reviewed Jul 2, 2024

View reviewed changes

mattzh72 pushed a commit that referenced this pull request Oct 9, 2024

feat: refactor CoreMemory to support generalized memory fields and …

8f46ab6

…memory editing functions (#1479) Co-authored-by: cpacker <[email protected]> Co-authored-by: Maximilian-Winter <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: refactor `CoreMemory` to support generalized memory fields and memory editing functions #1479

feat: refactor `CoreMemory` to support generalized memory fields and memory editing functions #1479

sarahwooders commented Jun 26, 2024 •

edited

Loading

Maximilian-Winter commented Jun 29, 2024 •

edited

Loading

sarahwooders commented Jul 1, 2024

cpacker Jul 1, 2024

cpacker Jul 1, 2024

cpacker Jul 1, 2024

cpacker Jul 1, 2024

sarahwooders Jul 1, 2024

cpacker left a comment

cpacker commented Jul 1, 2024

cpacker left a comment

cpacker commented Jul 1, 2024

norton120 left a comment

norton120 Jul 2, 2024

sarahwooders Jul 2, 2024

norton120 Jul 2, 2024

norton120 Jul 2, 2024

norton120 Jul 2, 2024

norton120 Jul 2, 2024

sarahwooders Jul 2, 2024

norton120 Jul 2, 2024

norton120 Jul 2, 2024

sarahwooders Jul 2, 2024

norton120 Jul 2, 2024

norton120 Jul 2, 2024

sarahwooders Jul 2, 2024

norton120 Jul 2, 2024

sarahwooders Jul 2, 2024 •

edited

Loading

norton120 Jul 2, 2024

feat: refactor CoreMemory to support generalized memory fields and memory editing functions #1479

feat: refactor CoreMemory to support generalized memory fields and memory editing functions #1479

Conversation

sarahwooders commented Jun 26, 2024 • edited Loading

Generalized Memory Class

Improve Agent Creation

Deprecated

Maximilian-Winter commented Jun 29, 2024 • edited Loading

sarahwooders commented Jul 1, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cpacker left a comment

Choose a reason for hiding this comment

cpacker commented Jul 1, 2024

cpacker left a comment

Choose a reason for hiding this comment

cpacker commented Jul 1, 2024

norton120 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sarahwooders Jul 2, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

feat: refactor `CoreMemory` to support generalized memory fields and memory editing functions #1479

feat: refactor `CoreMemory` to support generalized memory fields and memory editing functions #1479

sarahwooders commented Jun 26, 2024 •

edited

Loading

Maximilian-Winter commented Jun 29, 2024 •

edited

Loading

sarahwooders Jul 2, 2024 •

edited

Loading