Tools · SPBoucher
Menu

Crawlee

by Apify Technologies s.r.o. · Library · also known as Apify SDK (legacy name), crawlee-python

Open-source crawling and scraping library for Node.js/TypeScript and Python that unifies HTTP and headless-browser crawlers with queues, proxies, fingerprints and storage

Apache-2.0 licenseOpen sourceFreeLocalSelf-hostedCloudSelf-hostable
Visit official site Compare

Link verified Nov 2, 2026

Overview

Crawlee is Apify's open-source web crawling library, released under the Apache 2.0 license, available for Node.js and TypeScript and, since 2024, for Python. It provides a common crawler abstraction over different fetching strategies: plain HTTP crawlers with Cheerio or JSDOM parsing in JavaScript and BeautifulSoup or Parsel in Python, and browser crawlers driven by Playwright or Puppeteer, including an adaptive crawler that switches to a browser only when a page needs JavaScript.

Around that core it supplies the infrastructure most scrapers end up rebuilding: a persistent request queue with deduplication, automatic scaling based on system resources, retries and session management, proxy rotation, realistic browser fingerprints, and datasets and key-value stores with pluggable backends (file system, memory, SQL, Redis). Crawlers run anywhere Node or Python runs and deploy to the Apify platform as Actors with no code changes.

Crawlee suits developers who want Scrapy-like structure with first-class browser support and modern language tooling. The Python port is younger than the JavaScript library and has fewer integrations, and anti-bot evasion still depends on proxy quality.

Capabilities

  • Unified HTTP and browser crawler classes (Cheerio, JSDOM, Playwright, Puppeteer, BeautifulSoup, Parsel)
  • Persistent request queue with deduplication and autoscaling
  • Proxy rotation, sessions and browser fingerprinting
  • Datasets and key-value stores with multiple storage backends
  • Adaptive crawler that uses a browser only when needed
  • One-step deployment to the Apify platform

Typical workflows

  1. Crawl a documentation site to Markdown with an HTTP crawler
  2. Scrape a React-rendered catalog with PlaywrightCrawler
  3. Package a crawler as an Apify Actor

Strengths

  • Batteries included for queues, proxies and storage
  • Same mental model in JavaScript and Python
  • Apache 2.0 license and active development

Limitations

  • Python port is newer with fewer integrations
  • Browser crawling is resource-intensive
  • Evasion depends on external proxies

Prerequisites

  • Node.js 16 or newer, or Python 3.10 or newer

Who it is for

  • Developers building production scrapers
  • Teams deploying to Apify

Alternatives & comparison

  • Apify

    Cloud platform and marketplace for running web-scraping and automation programs (Actors), with storages, proxies, scheduling, an API and an MCP server for agents

    MixedFreemium
  • Browsertrix

    Webrecorder's browser-based web archiving system, available hosted or self-hosted, that crawls sites with real browsers into signed WACZ files replayable with ReplayWeb.page

    Open sourcePaid
  • Firecrawl

    Web data API that scrapes, crawls, maps and searches websites and returns clean Markdown or structured JSON for LLM pipelines, with an AGPL-licensed self-hostable core

    MixedFreemium
  • Playwright

    Microsoft's browser automation and end-to-end testing framework driving Chromium, Firefox and WebKit from one API in TypeScript, Python, Java and .NET

    Open sourceOpen source
  • Puppeteer

    Node.js library from the Chrome team for controlling Chrome and Firefox over the DevTools Protocol or WebDriver BiDi, used for scraping, screenshots, PDFs and testing

    Open sourceOpen source
  • Scrapy

    Mature open-source Python framework for writing crawlers (spiders) that fetch pages asynchronously, extract structured items and export them through pipelines

    Open sourceOpen source
Compare Crawlee with its alternatives →

Sources & verification

Primary sources consulted for this record. Claims are tied to the fields each source supports; the automated link checker re-verifies every URL on a schedule.

Confidence
High
Reviewed
Nov 1, 2026
Links
Verified · Nov 2, 2026

Spotted an error?

Broken link, outdated license, wrong category, or a tool that no longer exists? Tell us; reports are reviewed by the editor.

Firecrawl

Firecrawl (formerly Mendable) · Data collection & crawling

In-depth entry

Web data API that scrapes, crawls, maps and searches websites and returns clean Markdown or structured JSON for LLM pipelines, with an AGPL-licensed self-hostable core

AGPL-3.0-onlyFreemiumPaidSelf-hostedNov 2, 2026

Tavily

Tavily (Alpha AI Technologies Inc.) · Search & web-data APIs

In-depth entry

Web search, extraction, crawling and research API designed for LLM agents, returning concise source content with SDKs, framework integrations and a hosted MCP server

ProprietaryFreemiumUsage-basedNov 2, 2026

Zyte

Zyte Group Ltd · Data collection & crawling

Web-data company behind Scrapy offering Zyte API for ban-resistant fetching, browser rendering and automatic extraction, plus Scrapy Cloud for hosting spiders

ProprietaryUsage-basedPaidNov 2, 2026

Bright Data

Bright Data Ltd. · Data collection & crawling

Web data platform combining residential, datacenter and ISP proxy networks with unblocking, SERP and scraper APIs, remote browsers, ready-made datasets and an MCP server

ProprietaryFreemiumUsage-basedNov 2, 2026

Browsertrix

Webrecorder · Web archiving

Webrecorder's browser-based web archiving system, available hosted or self-hosted, that crawls sites with real browsers into signed WACZ files replayable with ReplayWeb.page

AGPL-3.0-onlyPaidEnterpriseSelf-hostedNov 2, 2026

Scrapfly

Scrapfly · Data collection & crawling

Web scraping API with anti-bot bypass, JavaScript rendering, cloud browsers, screenshots, AI extraction and a crawler, billed in credits per request

ProprietaryFreemiumPaidNov 2, 2026