Directory

Python framework for programming language-model pipelines with typed signatures and modules, then optimizing prompts and weights automatically against a metric

Open-source LLM observability and evaluation platform: tracing, LLM-as-judge and human evals, prompt management, datasets and cost dashboards, self-hostable or as Langfuse Cloud

Open-source Python SDK and self-hosted AI gateway that calls 100+ LLM providers in the OpenAI format, with routing, fallbacks, virtual keys, budgets, spend tracking and logging callbacks

Open-source, local-first CLI and library for evaluating prompts, models, RAG pipelines and agents, with automated red teaming and CI integration

Open-source Python library for evaluating LLM and RAG applications with LLM-driven and custom metrics, experiments, dataset management and result tracking