Directory

Hosted observability and evaluation platform for AI agents with tracing, experiments, pattern detection, a CLI, MCP access and an AI assistant (Loop) that analyzes traces and builds evals

Anthropic's developer platform for the Claude models: Messages API, tool use, structured outputs, prompt caching, batch processing and Managed Agents, plus cloud availability

Python framework for programming language-model pipelines with typed signatures and modules, then optimizing prompts and weights automatically against a metric

Google's developer API for the Gemini model family, keyed from Google AI Studio, with function calling, Search grounding, Live API, image and video generation and a free tier

Open-source LLM observability and evaluation platform: tracing, LLM-as-judge and human evals, prompt management, datasets and cost dashboards, self-hostable or as Langfuse Cloud

LangChain's hosted platform for tracing, evaluating and deploying LLM applications and agents, with prompt tooling, an LLM gateway and cloud, hybrid or self-hosted options

OpenAI's hosted developer platform: Responses and Chat Completions APIs, Realtime voice, embeddings, images, audio, fine-tuning, evals and managed agents

Unified API gateway to hundreds of models from many providers with automatic fallbacks, credit-based pass-through billing, BYOK, free models and data-policy routing

Open-source, local-first CLI and library for evaluating prompts, models, RAG pipelines and agents, with automated red teaming and CI integration

Type-safe Python agent framework from the Pydantic team with validated structured outputs, dependency injection, swappable model providers, MCP, durable execution and evals

Open-source Python library for evaluating LLM and RAG applications with LLM-driven and custom metrics, experiments, dataset management and result tracking

ML experiment tracking, sweeps, artifacts and model registry plus Weave for LLM tracing and evaluation, now sold by CoreWeave within its Forge platform with a free academic tier