Tools · SPBoucher
Menu

Common Crawl

by Common Crawl Foundation · Data service · also known as CC-MAIN, Common Crawl Foundation

Nonprofit open repository of petabyte-scale web crawl data since 2008, published as WARC, WAT and WET files on AWS Open Data with URL and columnar indexes

MixedFreeCloud
Visit official site Compare

Link verified Nov 2, 2026

Overview

Common Crawl is a nonprofit foundation (501(c)(3), founded in 2007) that crawls the web and publishes the results as an open corpus. Crawls are released several times a year, each labeled by period (for example CC-MAIN-2026-43), and each contains raw page captures in WARC files, metadata extracts (WAT) and plain-text extracts (WET). The data is hosted free of charge on Amazon Web Services' open data program and mirrored on some academic clouds; users process it in place with cloud tools or download selected segments.

To find pages, Common Crawl provides a URL index served through a CDX-style API and a columnar index in Parquet that can be queried with engines such as Athena, DuckDB or Spark. The corpus has become a foundational source for training language models and for research in web science, linguistics and information retrieval.

Access is free, subject to terms of use that place responsibility for third-party content and downstream uses, including AI training, on the user. Practical limits: each crawl is a sample rather than a complete snapshot, boilerplate and spam are abundant, a growing number of sites opt out, and working with the full dataset requires serious storage and compute.

Capabilities

  • Periodic web crawls in WARC, WAT and WET formats
  • Free hosting on AWS Open Data
  • URL index with CDX-style API
  • Columnar Parquet index for SQL querying
  • Open-source tooling on GitHub

Typical workflows

  1. Query the columnar index with DuckDB or Athena to select domains
  2. Extract text for a language corpus from WET files
  3. Study link structure or technology adoption across crawls

Strengths

  • Free and openly licensed access
  • Long time series since 2008
  • Standard archive formats and indexes

Limitations

  • Each crawl is a sample, not a full snapshot
  • Noisy content requiring heavy filtering
  • Large-scale processing costs fall on the user
  • Opt-outs reduce coverage over time

Prerequisites

  • Cloud or local compute able to read WARC/Parquet at scale

Who it is for

  • NLP and web-science researchers
  • Data engineers building corpora
  • Search and archive practitioners

Alternatives & comparison

  • Apify

    Cloud platform and marketplace for running web-scraping and automation programs (Actors), with storages, proxies, scheduling, an API and an MCP server for agents

    MixedFreemium
  • Bright Data

    Web data platform combining residential, datacenter and ISP proxy networks with unblocking, SERP and scraper APIs, remote browsers, ready-made datasets and an MCP server

    ProprietaryFreemium
  • Wayback Machine

    The Internet Archive's public web archive for browsing historical snapshots of web pages, with Save Page Now, Availability, CDX and Memento APIs

    ProprietaryFree
Compare Common Crawl with its alternatives →

Sources & verification

Primary sources consulted for this record. Claims are tied to the fields each source supports; the automated link checker re-verifies every URL on a schedule.

  • Official documentation
    Get Started

    https://commoncrawl.org/get-started

    Supports: capabilities, integrations

    checked Nov 1, 2026
  • Official documentation
    Common Crawl – Overview

    https://commoncrawl.org/overview

    Supports: capabilities, data_formats, description

    checked Nov 1, 2026
  • Official site
    Common Crawl

    https://commoncrawl.org/

    Supports: name, summary, vendor

    Homepage states 501(c)(3) non-profit founded in 2007

    checked Nov 1, 2026
  • Policy page
    Terms of Use

    https://commoncrawl.org/terms-of-use

    Supports: pricing_model, limitations

    checked Nov 1, 2026
Confidence
High
Reviewed
Nov 1, 2026
Links
Verified · Nov 2, 2026

Spotted an error?

Broken link, outdated license, wrong category, or a tool that no longer exists? Tell us; reports are reviewed by the editor.

Browsertrix

Webrecorder · Web archiving

Webrecorder's browser-based web archiving system, available hosted or self-hosted, that crawls sites with real browsers into signed WACZ files replayable with ReplayWeb.page

AGPL-3.0-onlyPaidEnterpriseSelf-hostedNov 2, 2026

ArchiveBox

ArchiveBox (Nick Sweeting and contributors) · Web archiving

Self-hosted, open-source web archiving tool that saves URLs as HTML, PDF, screenshot, WARC, readable text and media snapshots with a web UI and CLI

MITOpen sourceFreeLocalSelf-hostedNov 2, 2026