bug: Unexpected responses on regeneration

[phansel]

Jan version

v0.5.13

Describe the Bug

Llama 3.1 70B Instruct Q4 responds in an unexpected way after a response is removed and regenerated. It seems to be following up on the message that was regenerated, rather than regenerating from scratch.

It seems like this would happen if the deleted (regenerated) message is left in the chat log until the LM finishes sending the latest reply, rather than before it starts. Any follow-up questions posed after that last message get replies like "Hmm, it looks like I didn't answer your question originally."

Steps to Reproduce

Repro

1. Ask a question of the model
2. Observe reply
3. Regenerate the answer
4. Observe that the regenerated reply is not an answer to the question

Screenshots / Logs

Example

User 08:39:12 PM
Give me an airline's name.

Assistant 08:42:00 PM
Here is the name of an airline:
Lufthansa

User clicks "Regenerate" button on last message. The last message is replaced with this:

Assistant 08:47:35 PM
Would you like more information about Lufthansa or would you like another airline's name?

Pressing Regenerate on the last message again yields this:

Assistant 08:48:45 PM
Would you like another one?

What is your OS?

• MacOS
• Windows
• Linux

[imtuyethan]

This is a legacy issues, i had it many times in many versions but since it's model related so I didn't report.

[phansel]

This is a legacy issues, i had it many times in many versions but since it's model related so I didn't report.

Which models see it? The only other model I've tried (Llama 3.2 3B Instruct Q8) also shows it.

[louis-menlo]

@phansel @imtuyethan I also encountered an issue where regenerate sends the latest assistant message, which shouldn't, resulting in an incorrect question context. Additionally, regenerating anthropic models can sometimes lead to an empty response. These issues will be fixed in the next release.