[Roadmap] Logging and Observability

[sonichi]

We want the internal events in agents and conversation patterns to be logged. We also want to support integration with external loggers.
 

Tasks

Preview Give feedback

1. Track each llm request and response [PR Logging #1146]
2. Add telemetry ID to agent chat message
3. Logging for streaming mode
4. Logging for AgentEval
5. Add verbose level for the logging
6. Extra logging for teachable agent
7. Extra logging for group chat
8. Test logging with local models
9. Support Cost Calculation in V0.2 #770
   3 of 7
   0.2 feature +2
   
10. Profile cost and latency of test executio
11. [Feature Request]: Observability of agents' actions taken, tools used, and data accessed #2423
    0.2 logging needs-triage +3
12. feat: add event logging api and more tracing #2478
    +0
13. [Feature Request]: Support for custom loggers in runtime_logging #2581
    feature good first issue help wanted logging +4
    
14. How can I correspond the database data in runtime_logging with actual runtime outputs? #2670
    0.2 help wanted logging needs-triage +4

[sonichi]

After solving #203 logging can be done in a new way.

[victordibia]

This is related to #412 where we are discussing the structure of the content of the logged_history.
Given the overlap, I'll work on that when #203 is finalized.

[radman-x]

#203 appears to be done. When will we have logging in agents restored? This is VERY important.

[sonichi]

@victordibia it'll be nice to share progress if you know any or ask the dev working on this issue to share progress.

[victordibia]

@afourney , is there a branch we can link here on what Cheng is doing?
I agree with @sonichi that sharing progress will provide surface to provide feedback and also avoid duplicate work.

[afourney]

Not yet that I am aware of. Let's not develop anything here until we hear from her tomorrow

[cheng-tan]

Hi folks, I'm adding in the new logging for each agent, will have more updates this week.

[ekzhu]

@cheng-tan maybe create a draft PR so we are aware of this work item?

[cheng-tan]

Hi @radman-x, logging changes have been released with autogen v0.2.14 yesterday, documentation and example notebook. Please let me know if you have any questions or feedback, thanks.

[lalo]

I've been exploring with this notebook to consume the sqlite trace - sort of relevant to this issue: #2526

[Hk669]

i've made a PR on custom logger functionality, and the FileLogger #2596

[robinmanuelthiel]

Is it the plan to support Open Telemetry, especially for tracing?

[rysweet]

closing, please open separate issues for each unfinished item if you wish to reopen. 0.4 has Otel.