Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: added basic heartbeat override heuristics #621

Merged
merged 6 commits into from
Dec 25, 2023
Merged

Conversation

cpacker
Copy link
Collaborator

@cpacker cpacker commented Dec 15, 2023

Closes #601


Please describe the purpose of this pull request.

Local/open LLMs can often forget to request a heartbeat when running functions. E.g.:

> Enter your message: Using your archival memory, what do you know about Banh Mi?
💭 The user has asked me to use my archival knowledge to find information about 'Banh Mi'. This is a task I can handle easily. I will now search for all the memories containing that term.
⚡🧠 [function] updating memory with archival_memory_search
        query: banh mi, page: 0
> Enter your message:

This PR adds a basic heuristics to manually insert the heartbeat request:

  • If the user sends a message (type == user_message), and the agent immediately runs a function (non-send-message function), 99% chance the user is expecting a reply

How to test

  • Basic test parser (pytest -s -vvv tests/test_function_parser.py)
  • Test with old agents
  • Test with new agents
  • Test with OpenAI (regression)
  • Test with local LLM

Have you tested this PR?

Yes, see below.


Side note: had to patch a new bug in LM Studio where the prefix is no longer adhered to (probably to do with their default prompt formatting changing?), which requires unpacking a nested function due to our json patchers:

[2023-12-15 12:11:29.967] [INFO] [LM STUDIO SERVER] Generated prediction: {
  "id": "chatcmpl-",
  "object": "chat.completion",
  "created": 1702671084,
  "model": "/Users/loaner/lmstudio_models/TheBloke/OpenHermes-2.5-Mistral-7B-16k-GGUF/openhermes-2.5-mistral-7b-16k.Q8_0.gguf",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "{\n  \"function\": \"send_message\",\n  \"params\": {\n    \"inner_thoughts\": \"Activating persona Sam. Persona is designed to be curious, empathetic, and self-aware.\",\n    \"message\": \"Hello Chad! I'm Sam, your new companion. Nice to meet you!\"\n  }\n}"
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 2908,
    "completion_tokens": 71,
    "total_tokens": 2979
  }
}

@cpacker cpacker added the priority Merge ASAP label Dec 16, 2023
@cpacker cpacker changed the title added basic heartbeat override heuristics feat: added basic heartbeat override heuristics Dec 16, 2023
@cpacker
Copy link
Collaborator Author

cpacker commented Dec 25, 2023

Trying to patch out weird behavior on function returns:

what do you know about me?
actually my name is brad (short for bradley)
trigger name update

Example mixtral (hosted endpoint, temp = 0.8)

Before changes, model doesn't understand that the control got returned due to the heartbeat and goes off-topic / hallucinates:

image

Testing on mixtral (hosted endpoint, temp = 0.8)

New heartbeat str versions seem to be working

image

Testing on local dolphin 2.2.1

Also seems to be working

image

Testing on local openhermes 2.5

Bad func call but return is good

image

@cpacker cpacker merged commit 419ffc4 into main Dec 25, 2023
8 checks passed
@cpacker cpacker deleted the force-heartbeats branch December 25, 2023 07:46
sarahwooders pushed a commit that referenced this pull request Dec 26, 2023
* added basic heartbeat override

* tested and working on lmstudio (patched typo + patched new bug emerging in latest lmstudio build

* added lmstudio patch to chatml wrapper

* update the system messages to be informative about the source

* updated string constants after some tuning
norton120 pushed a commit to norton120/MemGPT that referenced this pull request Feb 15, 2024
* added basic heartbeat override

* tested and working on lmstudio (patched typo + patched new bug emerging in latest lmstudio build

* added lmstudio patch to chatml wrapper

* update the system messages to be informative about the source

* updated string constants after some tuning
mattzh72 pushed a commit that referenced this pull request Oct 9, 2024
* added basic heartbeat override

* tested and working on lmstudio (patched typo + patched new bug emerging in latest lmstudio build

* added lmstudio patch to chatml wrapper

* update the system messages to be informative about the source

* updated string constants after some tuning
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
priority Merge ASAP
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Manual heartbeat correction for local LLMs
1 participant