feat: Dynamic template system with editable system prompt fields.

[Maximilian-Winter]

Please describe the purpose of this pull request.
This implements a dynamic template field system for the system message editable at runtime through CLI. This also implements a customizable core memory without hard coded persona and human block. For use cases such as a customer service bot, dungeon master in a DnD game, where no persona and human is necessary or too unspecific and there is need for specific fields etc.

This all aims to make MemGPT more configurable for use cases outside that of a chatbot with a persona.

How to test
Load a system prompt with template fields and edit one through the cli. Create a customizable core memory preset.

More detail to the template field system for system message:
It adds a simple but effective template system for building the system message of memgpt.
It has predefined template fields that can be used to add parts of the default memgpt_chat system message.
The following shows the memgpt_chat system message in template form:

Each field adds a part of the system message of memgpt_chat, the template system will generate the memgpt system message automatically from it.

The template system get triggered when the system message string in the memgpt preset, ends with '_templated'. Like for example the following memgpt preset:

It then will load the built in template fields, like 'memgpt_introduction' or 'memgpt_realism_authenticity' and tries to load a file with the same name and path as the system message file, but having a file ending of '.yaml' consisting of field names with their respective content. The following is a complete example for instructing memgpt to talk like an old pirate:

memgpt_chat_pirate_templated.txt (in the user folder system_prompts)

{memgpt_introduction}

{memgpt_realism_authenticity}

{memgpt_control_flow}

{memgpt_basic_functions}

{memgpt_memory_editing}

{memgpt_recall_memory}

{memgpt_core_memory}

{memgpt_archival_memory}

{memgpt_introduction_end}

{pirate_instruction}

memgpt_chat_pirate_templated.yaml (in the user folder system_prompts)

pirate_instruction: 'Always reponsond like an old pirate!!!'

memgpt_pirate_preset.yaml (in the user folder presets)

system_prompt: "memgpt_chat_pirate_templated"
functions:
  - "send_message"
  - "pause_heartbeats"
  - "core_memory_append"
  - "core_memory_replace"
  - "conversation_search"
  - "conversation_search_date"
  - "archival_memory_insert"
  - "archival_memory_search"

Here is the resulting chat response of memgpt using the example preset:

The template fields are dynamic and editable through the CLI by using the "/edit_template_field" which prompts the user to input the template field name and its value. This allows to change parts of the system prompt dynamically during the runtime of the application.

Details on the configurable Core Memory
This makes it possible to initiate the core memory with other or more specific sections and information then the default persona and human section. To activate it the user has to specify his preset with a core_memory_type field set to "custom" and provide a custom_memory_file, which is an additional yaml containing sections and values of the initial core memory.

A user agent preset example:

system_prompt: "memgpt_chat"
core_memory_type: "custom"
core_memory_file: "initial_core_memory"
functions:
  - "send_message"
  - "pause_heartbeats"
  - "core_memory_append"
  - "core_memory_replace"
  - "conversation_search"
  - "conversation_search_date"
  - "archival_memory_insert"
  - "archival_memory_search"

The file inital_core_memory.yaml, next to the preset:

persona: 'The following is a blank slate starter persona, I need to expand this to develop my own personality.

My name is MemGPT.
I am kind, thoughtful, and inquisitive.'

human: 'First name: Chad'

This is just an example to create a default memgpt core memory with initial core memory. But you can put any fields and values you want as sections into the core memory yaml file.

Have you tested this PR?
Yes

Is your PR over 500 lines of code?
No

[Maximilian-Winter]

@cpacker @sarahwooders Maybe you both could give me a short feedback the next days on what you think about the system.

[cpacker]

This is really awesome @Maximilian-Winter , thank you for the PR! Left some high-level design comments (not necessarily requested changes, just some discussion points).

This looks roughly mergable to me, after some more discussion I can test and merge likely before v0.3 tag (or v0.3.1 latest).

[Maximilian-Winter]

@cpacker I have addressed all your comments by templating everything that make sense. I added:
system_message_layout_template
core_memory_section_template
To the agent preset. That will load the corresponding templates from the preset folder. If no template is specified it will fallback on the default template that is defined under presets/default_templates..

[Maximilian-Winter]

A full customized agent preset will look like this, it uses a templated system prompt, a custom core memory, a custom system_message_layout_template and custom core_memory_section_template:

system_prompt: "memgpt_chat_templated"
core_memory_type: "custom"
core_memory_file: "initial_core_memory"
system_message_layout_template: "system_message_layout_template_test"
core_memory_section_template: "core_memory_section_template_test"
functions:
  - "send_message"
  - "pause_heartbeats"
  - "core_memory_append"
  - "core_memory_replace"
  - "conversation_search"
  - "conversation_search_date"
  - "archival_memory_insert"
  - "archival_memory_search"

The MemGPT app will look in the preset folder for the two template files:
system_message_layout_template_test.txt
core_memory_section_template_test.txt

The fields:
core_memory_type
core_memory_file
system_message_layout_template
core_memory_section_template

Are not mandatory, but MemGPT will raise a exception if the field is defined and the file mentioned in it is not found.

I also changed the core_memory_file format to include the field limit, the following is a example

persona:
  content: 'The following is a blank slate starter persona, I need to expand this to develop my own personality.

My name is MemGPT.
I am kind, thoughtful, and inquisitive.'
  max_length: 150

human:
  content: 'First name: Chad'
  max_length: 150

The content field is mandatory, but max_length not, it will use a default value of 150 defined at creation of the customized core memory.

This is the default system_message_layout_template

{system}

### Memory [last modified: {memory_edit_timestamp}]
{len_recall_memory} previous messages between you and the user are stored in recall memory (use functions to access them)
{len_archival_memory} total memories you created are stored in archival memory (use functions to access them)

Core memory shown below (limited in size, additional information stored in archival / recall memory):
{core_memory_content}

This is the default core_memory_section_template:

<{memory_key} characters={memory_value_length}/{memory_value_limit}>
{memory_value}
</{memory_key}>

[tombedor]

I don't really get what the goal of this file is, or how it would be used

[Maximilian-Winter]

This is just a utility to regenerate the default template fields.

[tombedor]

this feels to me like an increase in complexity to a very core component of the overall agent.

I think the templated core memory feature is a cool idea, but I think the templating handling should be encapsulated into a class and then passed into this function.

Something like
def construct_system_with_memory(system: SystemPromptHandler, ...

then the full system message gets handled via

system.get_message() or something similar.

This way if someone wants to experiment with a different approach, they can simply swap out the SystemHandler class, rather than needing to work around the templating system here

[Maximilian-Winter]

I think you are right about that. I wrote the code after I hadn't slept for a night. 😄 And it was a wild idea.

[sarahwooders]

Hey @Maximilian-Winter - this is an awesome PR and we're really sorry about not merging it. We are currently working on generalizing the memory specification (and also will eventually generalize the templating). Given how much the codebase has changed since this PR, I think we'll re-implement some of this stuff in multiple individual PRs (the first of which is here #1479) and make you a co-author on those since I am looking at your implementation to inform the designs.

Again wanted to just let you know that although we didn't manage to merge this, @cpacker and I have looked over this PR very carefully and intent to implement parts of this -- so really appreciate your work.

[sarahwooders]

Closing since this is stale and we've incorporated into #1479 and #1584