fix: Fix handling of streaming response in AnthropicClaudeInvocationLayer

[silvanocerza]

Related Issues

• fixes Handle Anthropic streaming response #4990

Proposed Changes:

Change the default streaming handler in AnthropicClaudeInvocationLayer to a custom one, so that the received streamed response is handled gracefully.

How did you test it?

I added new tests for the AnthropicTokenStreamingHandler and changed the AnthropicClaudeInvocationLayer tests.

Notes for the reviewer

More info available in the linked issue.

[coveralls]

Pull Request Test Coverage Report for Build 5154996112

• 0 of 0 changed or added relevant lines in 0 files are covered.
• 2348 unchanged lines in 29 files lost coverage.
• Overall coverage increased (+0.6%) to 40.118%

---

Files with Coverage Reduction	New Missed Lines	%
nodes/file_converter/init.py	1	93.33%
agents/base.py	2	91.25%
init.py	2	83.33%
nodes/prompt/invocation_layer/handlers.py	2	93.88%
nodes/preprocessor/base.py	4	80.95%
utils/openai_utils.py	13	75.64%
nodes/prompt/invocation_layer/anthropic_claude.py	14	82.8%
utils/early_stopping.py	15	15.38%
nodes/answer_generator/openai.py	17	72.22%
nodes/prompt/prompt_template.py	20	89.9%

Totals
Change from base Build 5055949519:	0.6%
Covered Lines:	8994
Relevant Lines:	22419

---

💛 - Coveralls

[vblagoje]

@silvanocerza I suggest we make AnthropicTokenStreamingHandler an adapter class that takes in the user-providedTokenStreamingHandler via init. This instance of TokenStreamingHandler is what we pop from kwargs. We can then use AnthropicTokenStreamingHandler always - as an adapter; instead of print(chopped_text, flush=True, end="") we just invoke the user's TokenStreamingHandler with chopped text. This way, user-provided TokenStreamingHandler callbacks will get the same token-by-token callbacks across all invocation layers. For example, our agents send AgentTokenStreamingHandler as stream_handler kwarg down every invocation layer stack. In the currently proposed implementation, the user-provided TokenStreamingHandler (sent via kwargs as stream_handler) will get the old cumulative token callbacks, including our agents. Once Anthropic fixes their streaming, we can just retire AnthropicTokenStreamingHandler.

[vblagoje]

Almost there - if I'm not mistaken, the current version will not work as we want it to for user-provided stream_handler (e.g. our agent handler). We can adapt the proposed solution with a few lines of code and achieve our intended goal.

[silvanocerza]

Merged with branch fix-anthropic-streaming_v2.

[dfokina]

Updated docstrings language a bit

[vblagoje]

LGTM 🚀