Crawlers, scrapers, browser automation and extraction services.
13 tools
Filter in directory →Apify Technologies s.r.o. · Data collection & crawling
Cloud platform and marketplace for running web-scraping and automation programs (Actors), with storages, proxies, scheduling, an API and an MCP server for agents
Firecrawl (formerly Mendable) · Data collection & crawling
Web data API that scrapes, crawls, maps and searches websites and returns clean Markdown or structured JSON for LLM pipelines, with an AGPL-licensed self-hostable core
Microsoft · Testing & QA
Microsoft's browser automation and end-to-end testing framework driving Chromium, Firefox and WebKit from one API in TypeScript, Python, Java and .NET
Scrapy developers (maintained with Zyte) · Data collection & crawling
Mature open-source Python framework for writing crawlers (spiders) that fetch pages asynchronously, extract structured items and export them through pipelines
Tavily (Alpha AI Technologies Inc.) · Search & web-data APIs
Web search, extraction, crawling and research API designed for LLM agents, returning concise source content with SDKs, framework integrations and a hosted MCP server
Google (Chrome DevTools team) · Data collection & crawling
Node.js library from the Chrome team for controlling Chrome and Firefox over the DevTools Protocol or WebDriver BiDi, used for scraping, screenshots, PDFs and testing
Zyte Group Ltd · Data collection & crawling
Web-data company behind Scrapy offering Zyte API for ban-resistant fetching, browser rendering and automatic extraction, plus Scrapy Cloud for hosting spiders
Bright Data Ltd. · Data collection & crawling
Web data platform combining residential, datacenter and ISP proxy networks with unblocking, SERP and scraper APIs, remote browsers, ready-made datasets and an MCP server
Webrecorder · Web archiving
Webrecorder's browser-based web archiving system, available hosted or self-hosted, that crawls sites with real browsers into signed WACZ files replayable with ReplayWeb.page
Apify Technologies s.r.o. · Data collection & crawling
Open-source crawling and scraping library for Node.js/TypeScript and Python that unifies HTTP and headless-browser crawlers with queues, proxies, fingerprints and storage
Scrapfly · Data collection & crawling
Web scraping API with anti-bot bypass, JavaScript rendering, cloud browsers, screenshots, AI extraction and a crawler, billed in credits per request
SerpApi, LLC · Search & web-data APIs
API that returns live search-engine result pages from Google and other engines as structured JSON, with localization, device and search-type parameters
Jina AI · Data collection & crawling
Prefix-style API from Jina AI that converts any public URL into LLM-ready Markdown (r.jina.ai) or returns web search results with content (s.jina.ai), with open-source server code