External / Internal frameworks implementation (+ multi-agent)

[GALLLASMILAN]

Final suggestion [draft]

I would like to split the solution to the more parts and each part will have the priority

Common data

Here is a list of data (based on the openinference packages) we are able to collect from all of the external frameworks mentioned below.

attribute	example
traceId	(DM4tjgcj7C45km4kXGBmdw)
status	(error / ok)
framework	(langchain / dspy / crewai)
language	(Javascript / python)
input	(How are you today?)
output	(I am an agent, I am fine)

extended data

Some frameworks do not provide this data (like dspy)

attribute	example
provider	ollama
llm	llama3.1

framework specific data

see the whole span tree data for each framework and summaries I prepared to have full view on the provided data. Data are very often framework-specific.

framework	summary	all spans data
LangChain	openinference-langchain-data.txt	openinference-langchain-spans.json
CrewAI	openinference-crewai-data.txt	openinference-crewai-spans.json
Dspy	openinference-dspy-data.txt	openinference-dspy-spans.json
smolagents	openinference-smolagent-data.txt	openinference-smolagents-spans.json

I will add the more specific traces for the tool calling as well later.

Priority 1 = python frameworks

Openinference ✅

I found the openinference repo that contains a set of open-telemetry packages for a lot of our chosen frameworks like (crewAI, LangChain, and LangGraph) that we can use in our stack. The repositories are the technology independent and provide us with a solid telemetry solution. Otherwise, It would be so hard to create the telemetry solution for each framework from scratch.

• LLM Observability Semantic Conventions

IBM solution ⛔

• agent-analytics sdk
  • [openllmetry](base on https://github.com/traceloop/openllmetry) by traceloop
  • js version of openllmetry
• LLM Observability Semantic Conventions
  • LLM Semantic Conventions: Initial PR
  • APP

openllmetry ⛔

• [openllmetry](base on https://github.com/traceloop/openllmetry)
  There is support for crewAI and Langchain, but missing the support for autogen, dspy and smolagents.

Here is a list of the supported Python technologies we will use:

TODO: specify the supported frameworks in the first version

• CrewAI
  • openinference-instrumentation-crewai ✅
  • agent-analytics, opentelemetry_instrumentation_crewai = problems with installation. ✅
• Autogen
  • openinference-instrumentation-autogen => it's merged but not released yet. 🏗
  • agent-analytics = framework is missing ⛔
• Lang Chain
  • openinference
    • Python openinference-instrumentation-langchain ✅
    • Javascript ✅
  • agent-analytics
    • Python opentelemetry_instrumentation_langchain = Does not work and I cannot see any spans ⛔
    • Javascript = missing ⛔
• Dspy
  • openinference-instrumentation-dspy ✅
  • agent-analytics = framework is missing ⛔
• AWS labs
  • openinference
    • Python openinference-instrumentation-bedrock => only AWS labs agent observability -> I am not able to test it right now, but it should work ☘
  • agent-analytics = framework is missing ⛔
• smolagents
  • openinference-instrumentation-smolagents ✅
  • agent-analytics = framework is missing ⛔

---

agent creates the traceId
generate flowId

---

Call bee-agent-framework + observability

SINGLE agent

• tracing between javascript and Python languages ​​will be unified. In the python version of bee-hive, the emitter principle is already used. TODO: It will be necessary to provide the same trace interface -> so that the same thing is sent in both languages.
  @tomas.dvorak
• Single-agent support = We will use external frameworks (CrewAI, Atogen, AWS labs) and for each framework, we will need to make a "wrapper" that will compile events compatible with our framework from the internal emitter/log. For some frameworks, this will not even work, for example in AWS labs. => It will not work for those that do not support the emitter pattern (there will be more of these).
• In terms of compatibility of basic events, we will be able to provide most event names, but not data. So we will not be able to select specific data, such as rawPrompt now (an internal thing)

MULTI agent

• for support in the framework for "multi-agent systems" we will only support input/output in the first version. (for example -> now I started agent xx from crewUI, and now agent xx from crewAI has finished).
• "multi-agent approach" => We will support both (agent/workflow). This means that for example in aws-lab we can export a single agent and then use it within the workflow in bee, but we can also export the entire orchestration (multi-agent) and then use it in our bee. So we will have to provide instrumentation at both levels.

---

I found this project https://github.com/Arize-ai/openinference?tab=readme-ov-file where are opentelemerty providers for more frameworks like (crewAI, LangChain, DSPy, AWS Bedrock). I will inspire by this project

CrewAI

• docs telemetry

Instrumentation implementation (openinference)

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from openinference.instrumentation.crewai import CrewAIInstrumentor

endpoint = "http://127.0.0.1:4319/v1/traces"
trace_provider = TracerProvider()
trace.set_tracer_provider(trace_provider)
trace_provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint)))
CrewAIInstrumentor().instrument(tracer_provider=trace_provider)

What is very important, each span has the traceId property. It's very important

Crew AI span (openinference)

• input-span.json
• main.span.json

Crew AI span (agent-analytics)

• agent-analytics-trace.log

LangChain

• run with ollama
• observability

Instrumentation implementation

from opentelemetry import trace as trace_api
from opentelemetry.sdk import trace as trace_sdk
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace.export import ConsoleSpanExporter, BatchSpanProcessor
from openinference.instrumentation.langchain import LangChainInstrumentor

endpoint = "http://127.0.0.1:4319/v1/traces"
tracer_provider = trace_sdk.TracerProvider()
trace_api.set_tracer_provider(tracer_provider)
tracer_provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint)))
tracer_provider.add_span_processor(BatchSpanProcessor(ConsoleSpanExporter()))

LangChainInstrumentor().instrument()

LangChain span

• langchain-llm-event.json
• langchain-main-span.json

Autogen

AUTOGEN ANALYSIS 👁

• docs
• github repo
• Engine Agents node.js alternative. But it is not an official repo and it is not production ready.
• Ollama + Autogen + OpenAI compatibility => does not work 👎

Aws labs

Multi-Agent (awslabs) ANALYSIS

• aws labs
• github repo = python + typescript
• Ollama
• Logging in Multi-Agent Orchestrator

Dspy

• OpenInference DSPy Instrumentation
• dspy + ollama

Instrumentation implementation (openinference)

from openinference.instrumentation.dspy import DSPyInstrumentor
from opentelemetry import trace as trace_api
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk import trace as trace_sdk
from opentelemetry.sdk.trace.export import SimpleSpanProcessor

endpoint = "http://127.0.0.1:4319/v1/traces"
tracer_provider = trace_sdk.TracerProvider()
trace_api.set_tracer_provider(tracer_provider)
tracer_provider.add_span_processor(SimpleSpanProcessor(OTLPSpanExporter(endpoint)))

DSPyInstrumentor().instrument()

smolagents

• smolagents
• telemetry
• smolagents with ollama

Instrumentation implementation (openinference)

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

from openinference.instrumentation.smolagents import SmolagentsInstrumentor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

endpoint = "http://127.0.0.1:4319/v1/traces"
trace_provider = TracerProvider()
trace_provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint)))

SmolagentsInstrumentor().instrument(tracer_provider=trace_provider)

[GALLLASMILAN]

Hello @anafucs, I added some information about the collected data to the issue description. See the Common data, extended data and framework specific data sections.

I will continue with the data analytics tomorrow, so I may add more info there.

CC: @tomkis

[tomkis]

@anafucs please let us know if common + extended data is something that would work in your designs.

If you feel like there's some more info that would be good to visualise let us know and we'll investigate the option.

[GALLLASMILAN]

The next inspiration for us could be a phoenix observability tool that is part of the Arize-ai platform. The phoenix is based on the same data I analyzed from openinference packages.

I don't thing this tool is a good desing inspiration but it's good to know, how their native UI tool vizualize the data.

The don't pick the data and only vizualize them in 2 main forms.

• info = pretty view
• attibutes = json view

Then they have next 2 tabs for

• events = for example error
• feedback = human feedback (we don't support in our observe yet)

List

Trace error span detail on info tab

The trace span detail on attributes page

@anafucs @tomkis @mmurad2 @matoushavlena

[anafucs]

Thanks @GALLLASMILAN and @tomkis . we are thinking a solution very similar to the one above. On the common/extended data, the item I'm not sure is the language (as a primary info). This might be helpful in a drill down or when we add evaluation features (comparing agents)? We will share some drafts later today!

[GALLLASMILAN]

Hello @anafucs I only mentioned the language as an option and I don't think that is as important as others. (We can skip it).