Tools · SPBoucher
Menu

promptfoo

by Promptfoo, Inc. · CLI tool · also known as Promptfoo

Open-source, local-first CLI and library for evaluating prompts, models, RAG pipelines and agents, with automated red teaming and CI integration

MIT licenseOpen sourceFreeEnterpriseLocalSelf-hostedCloudSelf-hostable
Visit official site Compare

Checked Nov 2, 2026

Overview

promptfoo is an MIT-licensed command-line tool and Node library for test-driven development of LLM applications. A configuration file declares prompts, providers and test cases with assertions; the tool runs the matrix, scores outputs with deterministic checks or model-graded metrics, caches results and renders side-by-side comparisons in a local web viewer. It speaks to OpenAI, Anthropic, Azure, Google, Hugging Face, local runtimes and custom HTTP providers, and it runs in CI pipelines so prompt changes are gated like code. A second major use is red teaming: automated adversarial probes and compliance checks that produce vulnerability reports, plus related products for guardrails, model security scanning and an MCP proxy.

Evals execute on your machine and talk directly to the model provider, so no data passes through a hosted service unless you opt into sharing. The Community edition is free; Enterprise adds team sharing, SSO, continuous monitoring and managed cloud deployment, and an On-Premise edition deploys the same inside your infrastructure. promptfoo complements tracing platforms such as Langfuse rather than replacing them.

Capabilities

  • Declarative eval configs with prompts, providers and assertions
  • Deterministic and model-graded scoring with caching and concurrency
  • Side-by-side comparison in a local web viewer
  • Automated red teaming and vulnerability reports
  • Provider support for major APIs, local models and custom endpoints
  • CLI, Node library and CI integration

Typical workflows

  1. Gate prompt changes in CI with a regression suite
  2. Compare two models on the same test set
  3. Red-team a chatbot before launch

Strengths

  • Local-first; no data leaves your machine by default
  • MIT license
  • Covers both quality evals and security testing

Limitations

  • Not a production tracing backend
  • Model-graded assertions cost model calls
  • Enterprise collaboration features are proprietary

Prerequisites

  • Node.js
  • API keys or local models for the providers under test

Who it is for

  • Developers testing prompts and agents
  • Security teams red-teaming LLM apps

Alternatives & comparison

  • Braintrust

    Hosted observability and evaluation platform for AI agents with tracing, experiments, pattern detection, a CLI, MCP access and an AI assistant (Loop) that analyzes traces and builds evals

    MixedFreemium
  • DSPy

    Python framework for programming language-model pipelines with typed signatures and modules, then optimizing prompts and weights automatically against a metric

    Open sourceOpen source
  • Langfuse

    Open-source LLM observability and evaluation platform: tracing, LLM-as-judge and human evals, prompt management, datasets and cost dashboards, self-hostable or as Langfuse Cloud

    Open sourceOpen source
  • LangSmith

    LangChain's hosted platform for tracing, evaluating and deploying LLM applications and agents, with prompt tooling, an LLM gateway and cloud, hybrid or self-hosted options

    ProprietaryFreemium
  • Ragas

    Open-source Python library for evaluating LLM and RAG applications with LLM-driven and custom metrics, experiments, dataset management and result tracking

    Open sourceOpen source
Compare promptfoo with its alternatives →

Sources & verification

Primary sources consulted for this record. Claims are tied to the fields each source supports; the automated link checker re-verifies every URL on a schedule.

Confidence
High
Reviewed
Nov 1, 2026
Links
Broken · Nov 2, 2026

Spotted an error?

Broken link, outdated license, wrong category, or a tool that no longer exists? Tell us; reports are reviewed by the editor.

LiteLLM

BerriAI · Prompting, evaluation & observability

In-depth entry

Open-source Python SDK and self-hosted AI gateway that calls 100+ LLM providers in the OpenAI format, with routing, fallbacks, virtual keys, budgets, spend tracking and logging callbacks

MITOpen sourceFreeLocalSelf-hostedNov 2, 2026

OpenRouter

OpenRouter, Inc. · Model providers & APIs

Unified API gateway to hundreds of models from many providers with automatic fallbacks, credit-based pass-through billing, BYOK, free models and data-policy routing

ProprietaryFreemiumUsage-basedBroken

LangSmith

LangChain, Inc. · Prompting, evaluation & observability

LangChain's hosted platform for tracing, evaluating and deploying LLM applications and agents, with prompt tooling, an LLM gateway and cloud, hybrid or self-hosted options

ProprietaryFreemiumPaidSelf-hostedNov 2, 2026

Weights & Biases

CoreWeave · Training & fine-tuning

ML experiment tracking, sweeps, artifacts and model registry plus Weave for LLM tracing and evaluation, now sold by CoreWeave within its Forge platform with a free academic tier

MITFreemiumPaidSelf-hostedBroken