issue with tools + streaming + Claude on #133

[ultronozm]

Hi Andrew, I suspect this is still a work in progress, but figured I'd report what I come across:

Evaluating

(let* ((provider (make-llm-claude
                  :key (exec-path-from-shell-getenv "ANTHROPIC_KEY")
                  :chat-model "claude-3-5-sonnet-20240620"))
       results
       (add-fn (llm-make-tool-function
                :function (lambda (callback a b)
                           (let ((result (format "%s" (+ a b))))
                             (push (list :tool-call (cons (list a b) result)) results)
                             (funcall callback result)))
                :name "add"
                :description "Sums two numbers."
                :args '((:name "a" :description "A number." :type "integer" :required t)
                       (:name "b" :description "A number." :type "integer" :required t))
                :async t))
       (prompt (llm-make-chat-prompt
                "Compute 2+3."
                :tools (list add-fn)))
       done)
  (llm-chat-streaming
   provider prompt
   (lambda (partial) (push (list :partial partial) results))
   (lambda (final) 
     (push (list :final final) results)
     (push (list :prompt-after (copy-sequence prompt)) results)
     (setq done t))
   (lambda (err msg)
     (push (list :error err msg) results)
     (setq done t)))
  (while (not done) (sleep-for 0.1))
  (pp-display-expression (nreverse results) "*test*"))

yields

((:partial "Certainly! I can")
 (:partial "Certainly! I can help you compute the sum of 2 and 3 ")
 (:partial
  "Certainly! I can help you compute the sum of 2 and 3 using the \"add\" function that's available to us")
 (:partial
  "Certainly! I can help you compute the sum of 2 and 3 using the \"add\" function that's available to us. Let's use this tool to calculate the result.")
 (:final
  "Certainly! I can help you compute the sum of 2 and 3 using the \"add\" function that's available to us. Let's use this tool to calculate the result.")
 (:prompt-after
  #s(llm-chat-prompt nil nil
                     (#s(llm-chat-prompt-interaction user
                                                     "Compute 2+3."
                                                     nil)
                        #s(llm-chat-prompt-interaction assistant
                                                       "Certainly! I can help you compute the sum of 2 and 3 using the \"add\" function that's available to us. Let's use this tool to calculate the result."
                                                       nil))
                     (#s(llm-tool-function
                         #[(callback a b)
                           ((let ((result (format "%s" (+ a b))))
                              (setq results
                                    (cons
                                     (list :tool-call
                                           (cons (list a b) result))
                                     results))
                              (funcall callback result)))
                           ((results #1))]
                         "add" "Sums two numbers."
                         ((:name "a" :description "A number." :type
                                 "integer" :required t)
                          (:name "b" :description "A number." :type
                                 "integer" :required t))
                         t))
                     nil nil nil nil)))

which shows that when streaming with Claude,llm does not call the tool, let alone record the result in the prompt.

[ahyatt]

I mention this (briefly) in the README, but Claude doesn't support streaming tool use.

[ultronozm]

This was the case some time ago, but I believe the situation has changed. https://docs.anthropic.com/en/api/messages-streaming#streaming-request-with-tool-use

[ahyatt]

That's interesting, thanks. We'll have to update Claude's code to handle the streaming tool use. That will happen soon, but not in my current tool-use PR.

[ahyatt]

I did end up putting this in my latest PR, thanks for letting me know that Claude can now support this!

[ultronozm]

I tried this just now, or more precisely the mildly updated version

(let* ((provider (make-llm-claude
                  :key (exec-path-from-shell-getenv "ANTHROPIC_KEY")
                  :chat-model "claude-3-5-sonnet-20240620"))
       results
       (add-fn (llm-make-tool
                :function (lambda (callback a b)
                            (let ((result (format "%s" (+ a b))))
                              (push (list :tool-call (cons (list a b) result)) results)
                              (funcall callback result)))
                :name "add"
                :description "Sums two numbers."
                :args '((:name "a" :description "A number." :type integer :required t)
                        (:name "b" :description "A number." :type integer :required t))
                :async t))
       (prompt (llm-make-chat-prompt
                "Compute 2+3."
                :tools (list add-fn)))
       done)
  (llm-chat-streaming
   provider prompt
   (lambda (partial) (push (list :partial partial) results))
   (lambda (final) 
     (push (list :final final) results)
     (push (list :prompt-after (copy-sequence prompt)) results)
     (setq done t))
   (lambda (err msg)
     (push (list :error err msg) results)
     (setq done t)))
  (while (not done) (sleep-for 0.1))
  (pp-display-expression (nreverse results) "*test*"))

which failed for similar reasons as before. It seemed that you were calling the same function for streaming tool use for both OpenAI and Claude, but looking at their docs, they look different. Maybe I missed something. Anyway, I got tool streaming with Claude working on my branch, which I'll PR, referencing this, in just a second.

[ahyatt]

Thanks for the fix here - I'm not sure why I had thought I had enabled this. Perhaps it had worked on a simpler example. I've added an integration test so we know for sure now.