Python SDK for AI agent monitoring, LLM cost tracking, benchmarking, and more. Integrates with most LLMs and agent frameworks like CrewAI, Langchain, and Autogen

🥤 RAGLite is a Python toolkit for Retrieval-Augmented Generation (RAG) with PostgreSQL or SQLite

A library for evaluating Retrieval-Augmented Generation (RAG) systems (The traditional ways).

Evalica, your favourite evaluation toolkit

Our curated collection of templates. Use these patterns to set up your AI projects for evaluation with Openlayer.

An implementation of the Anthropic's paper and essay on "A statistical approach to model evaluations"

Develop better LLM apps by testing different models and prompts in bulk.

The Modelmetry Python SDK allows developers to easily integrate Modelmetry’s advanced guardrails and monitoring capabilities into their LLM-powered applications.