Directory

Mature open-source Python framework for writing crawlers (spiders) that fetch pages asynchronously, extract structured items and export them through pipelines

API that returns live search-engine result pages from Google and other engines as structured JSON, with localization, device and search-type parameters

Python documentation generator using reStructuredText or MyST Markdown with cross-referencing, API autodoc and output to HTML, PDF via LaTeX, ePub and man pages

Web search, extraction, crawling and research API designed for LLM agents, returning concise source content with SDKs, framework integrations and a hosted MCP server

Cloud platform for open-source models: serverless and dedicated inference, fine-tuning, GPU clusters, batch jobs and code sandboxes behind an OpenAI-compatible API

Hosted embedding and reranking models for retrieval, including code, finance, law and multimodal variants, offered by MongoDB with a free token allowance and pay-as-you-go API

ML experiment tracking, sweeps, artifacts and model registry plus Weave for LLM tracing and evaluation, now sold by CoreWeave within its Forge platform with a free academic tier

Institution-subscribed research data platform from Wharton hosting CRSP, Compustat, TAQ, OptionMetrics and dozens of other vendors' databases with web queries, SQL, SAS, Python and R access

Web-data company behind Scrapy offering Zyte API for ban-resistant fetching, browser rendering and automatic extraction, plus Scrapy Cloud for hosting spiders