Tools · SPBoucher
Menu

Scrapy

In-depth entry

by Scrapy developers (maintained with Zyte) · Framework

Mature open-source Python framework for writing crawlers (spiders) that fetch pages asynchronously, extract structured items and export them through pipelines

BSD-3-Clause licenseOpen sourceFreeLocalSelf-hostedCloudSelf-hostable
Visit official site Compare

Link verified Nov 2, 2026

Overview

Scrapy is an open-source Python framework for crawling websites and extracting structured data, in development since 2008 and released under the BSD 3-Clause license. Its development has long been associated with the company now called Zyte, which also offers hosted Scrapy Cloud, but the framework itself is community-governed on GitHub.

A Scrapy project defines spiders, classes that declare start URLs and parse callbacks; the engine schedules requests through an asynchronous downloader built on Twisted, applies downloader and spider middlewares (for headers, retries, proxies, cookies, throttling), and routes extracted items through item pipelines for validation, deduplication and storage. Selectors use XPath and CSS, item loaders normalize fields, and feed exports write JSON, JSON Lines, CSV or XML to local files or cloud storage. Built-in features include auto-throttle, robots.txt handling, caching, link extraction, a shell for interactive testing and a stats collector. Extensions add Playwright integration for JavaScript-heavy sites and deployment via Scrapyd or Scrapy Cloud.

Scrapy suits data engineers and researchers who need repeatable, large-scale crawls of static or server-rendered sites with clean separation between fetching, parsing and storage. Current releases require Python 3.10 or newer.

Limits: it does not render JavaScript without an add-on, its Twisted-based architecture and project structure impose a learning curve, and anti-bot evasion, proxy pools and distributed crawling are left to extensions or commercial services.

Capabilities

  • Asynchronous crawling engine with request scheduling
  • Spiders with XPath and CSS selectors and item loaders
  • Downloader and spider middlewares for retries, proxies and throttling
  • Item pipelines for cleaning and storage
  • Feed exports to JSON, CSV, XML and cloud storage
  • Interactive shell, caching and crawl statistics
  • Deployment through Scrapyd and Scrapy Cloud

Typical workflows

  1. Crawl a listings site nightly and export JSON Lines
  2. Build a focused crawler for a research corpus
  3. Add Playwright to a spider for client-rendered pages
  4. Schedule spiders on Scrapyd or Scrapy Cloud

Strengths

  • Long-lived, well-documented framework
  • Clear architecture separating fetch, parse and store
  • Permissive BSD license
  • Rich extension ecosystem

Limitations

  • No JavaScript rendering without extensions
  • Learning curve from Twisted and project structure
  • Anti-bot and proxy handling not included
  • Python only

Prerequisites

  • Python 3.10 or newer

Who it is for

  • Data engineers building crawl pipelines
  • Researchers collecting web corpora
  • Developers of focused crawlers

Alternatives & comparison

Crawlee is the nearest modern equivalent, available for Node.js and Python, with browser crawlers, proxy rotation and fingerprinting built in; Scrapy is older, HTTP-first and more widely deployed in Python shops. Puppeteer and Playwright are browser-automation libraries rather than crawl frameworks, usually combined with Scrapy or Crawlee for JavaScript-rendered pages. Apify is a hosted platform where Scrapy spiders can be packaged as Actors, adding scheduling, storage and proxies.

  • Apify

    Cloud platform and marketplace for running web-scraping and automation programs (Actors), with storages, proxies, scheduling, an API and an MCP server for agents

    MixedFreemium
  • Crawlee

    Open-source crawling and scraping library for Node.js/TypeScript and Python that unifies HTTP and headless-browser crawlers with queues, proxies, fingerprints and storage

    Open sourceOpen source
  • Playwright

    Microsoft's browser automation and end-to-end testing framework driving Chromium, Firefox and WebKit from one API in TypeScript, Python, Java and .NET

    Open sourceOpen source
  • Puppeteer

    Node.js library from the Chrome team for controlling Chrome and Firefox over the DevTools Protocol or WebDriver BiDi, used for scraping, screenshots, PDFs and testing

    Open sourceOpen source
  • Zyte

    Web-data company behind Scrapy offering Zyte API for ban-resistant fetching, browser rendering and automatic extraction, plus Scrapy Cloud for hosting spiders

    ProprietaryUsage-based
Compare Scrapy with its alternatives →

Sources & verification

Primary sources consulted for this record. Claims are tied to the fields each source supports; the automated link checker re-verifies every URL on a schedule.

Confidence
High
Reviewed
Nov 1, 2026
Links
Verified · Nov 2, 2026

Spotted an error?

Broken link, outdated license, wrong category, or a tool that no longer exists? Tell us; reports are reviewed by the editor.

Firecrawl

Firecrawl (formerly Mendable) · Data collection & crawling

In-depth entry

Web data API that scrapes, crawls, maps and searches websites and returns clean Markdown or structured JSON for LLM pipelines, with an AGPL-licensed self-hostable core

AGPL-3.0-onlyFreemiumPaidSelf-hostedNov 2, 2026

Tavily

Tavily (Alpha AI Technologies Inc.) · Search & web-data APIs

In-depth entry

Web search, extraction, crawling and research API designed for LLM agents, returning concise source content with SDKs, framework integrations and a hosted MCP server

ProprietaryFreemiumUsage-basedNov 2, 2026

Zyte

Zyte Group Ltd · Data collection & crawling

Web-data company behind Scrapy offering Zyte API for ban-resistant fetching, browser rendering and automatic extraction, plus Scrapy Cloud for hosting spiders

ProprietaryUsage-basedPaidNov 2, 2026

Bright Data

Bright Data Ltd. · Data collection & crawling

Web data platform combining residential, datacenter and ISP proxy networks with unblocking, SERP and scraper APIs, remote browsers, ready-made datasets and an MCP server

ProprietaryFreemiumUsage-basedNov 2, 2026

Browsertrix

Webrecorder · Web archiving

Webrecorder's browser-based web archiving system, available hosted or self-hosted, that crawls sites with real browsers into signed WACZ files replayable with ReplayWeb.page

AGPL-3.0-onlyPaidEnterpriseSelf-hostedNov 2, 2026

Scrapfly

Scrapfly · Data collection & crawling

Web scraping API with anti-bot bypass, JavaScript rendering, cloud browsers, screenshots, AI extraction and a crawler, billed in credits per request

ProprietaryFreemiumPaidNov 2, 2026