Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: improve smaller model response on user refusals #374

Merged

Conversation

jrmi
Copy link
Contributor

@jrmi jrmi commented Dec 31, 2024

I noticed that when you refuse the operation when asked for confirmation, small models (at least gpt-o4-mini) are a bit lost and try again the same command without any modification multiple times which make the confirmation useless.

To test it, with a gpt-o4-mini just use a prompt like "Create the file foo.txt with random content.and sayNo` when the tool ask for confirmation to save and the mode will call the tool again with the same parameters.

In this MR I tweaked a bit the initial prompt and the response message in these cases to prevent that behavior.


Important

Improves response behavior for smaller models by updating prompts and messages to prevent redundant command execution after user refusals.

  • Behavior:
    • Updates confirmation messages in chat.py, commands.py, tools/patch.py, tools/python.py, tools/save.py, tools/tmux.py, and util/ask_execute.py to specify user-initiated interruptions and refusals.
    • Modifies prompt_gptme() in prompts.py to instruct the model not to retry operations after user refusal and to ask for clarification instead.
  • Prompts:
    • Adds a line in prompt_gptme() to handle user-aborted operations by asking for clarification rather than retrying.
  • Messages:
    • Changes various system messages to clarify when a user has aborted an operation, e.g., "User hit Ctrl-c to interrupt the process" and "Aborted, user chose not to run this code."

This description was created by Ellipsis for 3f54a02. It will automatically update as commits are pushed.

@jrmi jrmi changed the title fix: improve smaller model response for user refusals fix: improve smaller model response on user refusals Dec 31, 2024
Copy link
Contributor

@ellipsis-dev ellipsis-dev bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 Looks good to me! Reviewed everything up to 3f54a02 in 27 seconds

More details
  • Looked at 146 lines of code in 8 files
  • Skipped 0 files when reviewing.
  • Skipped posting 7 drafted comments based on config settings.
1. gptme/chat.py:141
  • Draft comment:
    Improved message clarity by specifying the user action. This change is consistent with the PR's goal to enhance user feedback when operations are aborted.
  • Reason this comment was not posted:
    Confidence changes required: 10%
    The PR aims to improve the response when a user refuses an operation. The changes are consistent across multiple files, ensuring that the system message is more descriptive when a user aborts an operation. This is a good practice for clarity and user feedback.
2. gptme/commands.py:186
  • Draft comment:
    Improved message clarity by specifying the user action. This change is consistent with the PR's goal to enhance user feedback when operations are aborted.
  • Reason this comment was not posted:
    Confidence changes required: 10%
    The PR aims to improve the response when a user refuses an operation. The changes are consistent across multiple files, ensuring that the system message is more descriptive when a user aborts an operation. This is a good practice for clarity and user feedback.
3. gptme/tools/patch.py:244
  • Draft comment:
    Improved message clarity by specifying the user action. This change is consistent with the PR's goal to enhance user feedback when operations are aborted.
  • Reason this comment was not posted:
    Confidence changes required: 10%
    The PR aims to improve the response when a user refuses an operation. The changes are consistent across multiple files, ensuring that the system message is more descriptive when a user aborts an operation. This is a good practice for clarity and user feedback.
4. gptme/tools/python.py:117
  • Draft comment:
    Improved message clarity by specifying the user action. This change is consistent with the PR's goal to enhance user feedback when operations are aborted.
  • Reason this comment was not posted:
    Confidence changes required: 10%
    The PR aims to improve the response when a user refuses an operation. The changes are consistent across multiple files, ensuring that the system message is more descriptive when a user aborts an operation. This is a good practice for clarity and user feedback.
5. gptme/tools/save.py:100
  • Draft comment:
    Improved message clarity by specifying the user action. This change is consistent with the PR's goal to enhance user feedback when operations are aborted.
  • Reason this comment was not posted:
    Confidence changes required: 10%
    The PR aims to improve the response when a user refuses an operation. The changes are consistent across multiple files, ensuring that the system message is more descriptive when a user aborts an operation. This is a good practice for clarity and user feedback.
6. gptme/tools/tmux.py:173
  • Draft comment:
    Improved message clarity by specifying the user action. This change is consistent with the PR's goal to enhance user feedback when operations are aborted.
  • Reason this comment was not posted:
    Confidence changes required: 10%
    The PR aims to improve the response when a user refuses an operation. The changes are consistent across multiple files, ensuring that the system message is more descriptive when a user aborts an operation. This is a good practice for clarity and user feedback.
7. gptme/util/ask_execute.py:250
  • Draft comment:
    Improved message clarity by specifying the user action. This change is consistent with the PR's goal to enhance user feedback when operations are aborted.
  • Reason this comment was not posted:
    Confidence changes required: 10%
    The PR aims to improve the response when a user refuses an operation. The changes are consistent across multiple files, ensuring that the system message is more descriptive when a user aborts an operation. This is a good practice for clarity and user feedback.

Workflow ID: wflow_B2AtNfXGSnMSsDmq


You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.

@jrmi jrmi force-pushed the fix-user-refusal-for-smaller-models branch from 3f54a02 to 49a2110 Compare December 31, 2024 10:33
@jrmi jrmi force-pushed the fix-user-refusal-for-smaller-models branch from 49a2110 to 3ade688 Compare January 4, 2025 15:19
@ErikBjare
Copy link
Owner

ErikBjare commented Jan 4, 2025

Nice, been meaning to do this for a while. Even confuses Sonnet sometimes :)

@ErikBjare ErikBjare merged commit 11708dd into ErikBjare:master Jan 4, 2025
5 of 7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants