Tools · SPBoucher
Menu

Firecrawl

In-depth entry

by Firecrawl (formerly Mendable) · API service · also known as mendableai/firecrawl

Web data API that scrapes, crawls, maps and searches websites and returns clean Markdown or structured JSON for LLM pipelines, with an AGPL-licensed self-hostable core

AGPL-3.0-only licenseFreemiumPaidUsage-basedEnterpriseCloudSelf-hostedSelf-hostable
Visit official site Compare

Link verified Nov 2, 2026

Overview

Firecrawl is an API service for turning websites into data that language models can consume. It began as an open-source project from the Mendable team (the GitHub repository moved from mendableai to the firecrawl organization) and is now a hosted product with an AGPL-3.0-licensed core that can also be self-hosted.

The API is organized around a few verbs. Scrape fetches one URL and returns Markdown, HTML, links, screenshots, metadata or JSON extracted against a schema; Crawl follows a site recursively and collects every page; Map lists a site's URLs quickly; Search runs a web query and optionally returns page content for each hit; Interact continues a scrape session to click, fill forms and read dynamic content; Parse converts local files such as PDF and Office documents; and an Agent mode gathers data autonomously. The service handles JavaScript rendering, proxies and anti-bot measures and delivers asynchronous results through webhooks. Official SDKs exist for Python and Node.js alongside a CLI, a hosted MCP server for coding agents, and integrations with LangChain, LlamaIndex and other frameworks.

Firecrawl is popular for building retrieval corpora, feeding agents with current web content and monitoring documentation sites. Pricing is credit-based with a free tier, subscription plans, pay-as-you-go top-ups and enterprise agreements.

Limits: the hosted service is proprietary beyond the open core and self-hosting lacks some managed features, credit consumption varies by endpoint and option, and like any scraper it cannot access content that sites block or that requires authenticated sessions it does not have.

Capabilities

  • Scrape a URL to Markdown, HTML, links, screenshots or schema-based JSON
  • Recursive site crawl with asynchronous jobs and webhooks
  • Fast URL discovery with Map
  • Web search with optional page content
  • Interactive sessions for clicks and form filling
  • Document parsing for PDF and Office files
  • Python and Node.js SDKs, CLI and MCP server
  • Self-hostable open-source core

Typical workflows

  1. Build a RAG corpus from a documentation site
  2. Feed an agent live web pages in Markdown
  3. Monitor a set of pages for changes
  4. Extract product fields into JSON with a schema

Strengths

  • Output tailored to LLM consumption
  • One API for scrape, crawl, map and search
  • Open core allows self-hosting
  • Broad framework and MCP integrations

Limitations

  • Hosted features exceed what the self-hosted core offers
  • Credit costs vary by endpoint and option
  • Cannot bypass sites that block or require logins
  • AGPL obligations apply when modifying and serving the core

Prerequisites

  • API key for the hosted service
  • Docker for self-hosting

Who it is for

  • Developers building RAG and agent pipelines
  • Teams needing clean text from many sites

Alternatives & comparison

Jina Reader covers the single-URL-to-Markdown case with a simpler prefix-style API and a search endpoint, without full crawl orchestration. Tavily is primarily a search and extraction API for agents, overlapping on search and extract but not on recursive crawling. Apify offers a marketplace of site-specific scrapers and hosted Actors rather than a generic page-to-Markdown API. Scrapfly is a scraping API centred on anti-bot bypass and raw HTML; Crawlee is an open-source library you run and operate yourself.

  • Apify

    Cloud platform and marketplace for running web-scraping and automation programs (Actors), with storages, proxies, scheduling, an API and an MCP server for agents

    MixedFreemium
  • Crawlee

    Open-source crawling and scraping library for Node.js/TypeScript and Python that unifies HTTP and headless-browser crawlers with queues, proxies, fingerprints and storage

    Open sourceOpen source
  • Jina Reader

    Prefix-style API from Jina AI that converts any public URL into LLM-ready Markdown (r.jina.ai) or returns web search results with content (s.jina.ai), with open-source server code

    MixedFreemium
  • Scrapfly

    Web scraping API with anti-bot bypass, JavaScript rendering, cloud browsers, screenshots, AI extraction and a crawler, billed in credits per request

    ProprietaryFreemium
  • Tavily

    Web search, extraction, crawling and research API designed for LLM agents, returning concise source content with SDKs, framework integrations and a hosted MCP server

    ProprietaryFreemium
Compare Firecrawl with its alternatives →

Sources & verification

Primary sources consulted for this record. Claims are tied to the fields each source supports; the automated link checker re-verifies every URL on a schedule.

Confidence
High
Reviewed
Nov 1, 2026
Links
Verified · Nov 2, 2026

Spotted an error?

Broken link, outdated license, wrong category, or a tool that no longer exists? Tell us; reports are reviewed by the editor.

Playwright

Microsoft · Testing & QA

In-depth entry

Microsoft's browser automation and end-to-end testing framework driving Chromium, Firefox and WebKit from one API in TypeScript, Python, Java and .NET

Apache-2.0Open sourceFreeLocalSelf-hostedNov 2, 2026

Scrapy

Scrapy developers (maintained with Zyte) · Data collection & crawling

In-depth entry

Mature open-source Python framework for writing crawlers (spiders) that fetch pages asynchronously, extract structured items and export them through pipelines

BSD-3-ClauseOpen sourceFreeLocalSelf-hostedNov 2, 2026

Puppeteer

Google (Chrome DevTools team) · Data collection & crawling

In-depth entry

Node.js library from the Chrome team for controlling Chrome and Firefox over the DevTools Protocol or WebDriver BiDi, used for scraping, screenshots, PDFs and testing

Apache-2.0Open sourceFreeLocalSelf-hostedNov 2, 2026

Zyte

Zyte Group Ltd · Data collection & crawling

Web-data company behind Scrapy offering Zyte API for ban-resistant fetching, browser rendering and automatic extraction, plus Scrapy Cloud for hosting spiders

ProprietaryUsage-basedPaidNov 2, 2026

Bright Data

Bright Data Ltd. · Data collection & crawling

Web data platform combining residential, datacenter and ISP proxy networks with unblocking, SERP and scraper APIs, remote browsers, ready-made datasets and an MCP server

ProprietaryFreemiumUsage-basedNov 2, 2026

Browsertrix

Webrecorder · Web archiving

Webrecorder's browser-based web archiving system, available hosted or self-hosted, that crawls sites with real browsers into signed WACZ files replayable with ReplayWeb.page

AGPL-3.0-onlyPaidEnterpriseSelf-hostedNov 2, 2026