Scraping And Data Extraction Resources
April 29, 2026 · View on GitHub
A curated index of tools, libraries, frameworks, and utilities for scraping web data — with a focus on packaging extracted data for consumption by AI agents (RAG pipelines, LLM context, agent tools).
Snapshot: 2026-04-29
Each entry uses dynamic badges for ⭐ star count and last-updated date — the values refresh automatically when GitHub state changes.
Scope
This index covers:
- Scrapers & crawlers — headless browser automation, HTTP scrapers, site mirrors
- Extraction & parsing — HTML → structured data, readability/article extractors, markdown converters
- AI-ready packaging — LLM-optimized scrapers and converters that output Markdown, JSON, or chunked text suitable for embeddings, RAG ingestion, or direct LLM context
- Agent integrations — MCP servers exposing scraping/extraction as tools to LLM agents
- Anti-bot & rendering — JS rendering, stealth/fingerprinting, proxy rotation utilities
For headless-browser automation aimed at agentic browsing (rather than data extraction), see the related index AI-Browser-Tools.
Table of Contents
- LLM-Optimized Scrapers
- MCP Servers
- Frameworks / Tools
- Extractors & Parsers
- Markdown Conversion
- Document & Component Extraction
- Mobile
- Use Cases & Recipes
- Israel-Specific
- Awesome Lists
- Related Indexes
LLM-Optimized Scrapers
Scrapers designed to feed LLMs directly — schema-driven extraction, structured output, AI-friendly formats.
llm-scraper
Turn any webpage into structured data using LLMs (Node.js, schema-driven via Zod).
scrapeghost
Experimental Python library for scraping websites using OpenAI's GPT — pass a URL and a schema, get structured data.
scrapecraft
Visual builder for ScrapeGraphAI agentic scraping pipelines.
firecrawl
Production-grade web crawler that turns sites into LLM-ready Markdown / structured data — the engine behind the Firecrawl service.
Scrapegraph-ai
Python library for LLM-driven web scraping — describe what you want in natural language, get structured data.
crawl4ai
LLM-friendly web crawler with adaptive extraction, JS rendering, and Markdown output.
gpt-crawler
Crawl a site and produce a single knowledge file you can drop into a custom GPT or RAG pipeline.
llmsherpa
Layout-aware document extraction for LLMs — preserves sections, tables, and lists from PDFs/HTML.
MCP Servers
MCP servers that expose scraping, crawling, or markdown-conversion capabilities to AI agents.
markdown-downloader
MCP server that downloads webpages and converts them to Markdown for agent consumption.
markdownify-mcp
MCP server for converting files and web content (PDFs, images, audio, YouTube, web pages) into Markdown.
firecrawl-mcp-server
Official MCP server for Firecrawl — web crawling, scraping, and structured extraction.
firecrawl/firecrawl-mcp-server
scrapegraph-mcp
Official MCP server for ScrapeGraphAI — agentic, schema-driven web scraping.
supadata mcp
MCP server for Supadata — structured data extraction from web, YouTube, and documents.
mcp-read-website-fast
Fast, lightweight MCP server for reading webpages and returning clean Markdown.
just-every/mcp-read-website-fast
webclaw
MCP server for web scraping and content extraction.
linkedin-mcp-server
MCP server for scraping LinkedIn profiles and data.
eliasbiondo/linkedin-mcp-server
crawl4ai-mcp-server
MCP server wrapping Crawl4AI — LLM-friendly web crawler with adaptive extraction.
crawlbase-mcp
Official MCP server for the Crawlbase scraping API.
mcp-web-scrape
MCP server exposing web-scraping tools to LLM agents.
mcp-server-docy
MCP server for fetching and serving documentation pages to agents.
scraper-mcp
MCP server wrapping a general-purpose web scraper.
browsegenie
Browser-driven scraping MCP — agent-controlled page extraction.
opencode-browser
Browser tool integration for OpenCode — fetch and extract page content for agent context.
markitdown_mcp_server
MCP server wrapping Microsoft's markitdown — convert files and URLs to Markdown for LLM consumption.
KorigamiK/markitdown_mcp_server
scrapling-fetch-mcp
MCP server wrapping Scrapling — undetectable, adaptive page fetching exposed as an agent tool.
cyberchitta/scrapling-fetch-mcp
html-to-markdown-mcp
MCP server that converts arbitrary HTML to clean Markdown — ideal for piping scraped pages into LLM context.
Frameworks / Tools
General-purpose scraping frameworks and libraries across languages.
scrapy
The canonical Python framework for large-scale web crawling and scraping.
ferret
Declarative, query-language-based web scraping (FQL — Ferret Query Language) in Go.
Scrapling
Undetectable, adaptive Python scraping library — auto-heals broken selectors when sites change.
rod
High-level Go wrapper around the Chrome DevTools Protocol — headless browser scraping.
jsoup
Java library for working with real-world HTML — parsing, manipulation, CSS-selector extraction.
scraper (Rust)
HTML parsing and querying with CSS selectors for Rust.
scala-scraper
Scala library for scraping content from HTML pages.
scrape
Simple, command-line web scraper for Go.
linkedin_scraper
Python scraper for LinkedIn profiles, companies, and jobs.
facebook-scraper
Python package for scraping public Facebook posts without API access.
obscura
Stealth scraping toolkit aimed at evading bot detection.
XActions
Scraper / automation toolkit for X (Twitter) — extract posts, threads, and profile data.
crawlee
Production-grade Node.js crawler framework — handles queueing, retries, proxies, and storage.
crawlee-python
Python port of Crawlee — full-featured crawler framework with HTTP and browser-based fetchers.
requests-html
Pythonic, high-level HTTP scraping library with built-in HTML parsing and JS rendering.
MechanicalSoup
Python library for automating interaction with websites — wraps Requests and BeautifulSoup.
Extractors & Parsers
Targeted extraction of metadata, articles, and structured fields.
metascraper
Library for extracting structured metadata (title, author, image, publication date) from any URL — rule-based.
trafilatura
Best-in-class Python library for web text and metadata extraction — article body, comments, publication date, language.
readability
Mozilla's standalone version of the Reader Mode library — turns cluttered pages into clean article text.
goose3
Article extractor for Python — pulls main text, top image, and metadata from news pages.
newspaper
Article scraping and curation for Python (newspaper3k) — author, publish date, top image, full text.
cheerio
jQuery-style server-side HTML parsing and manipulation for Node.js — fast and lean.
lxml
Fast, feature-rich Python library for processing XML and HTML — backbone of many scrapers.
Markdown Conversion
Markdown parsers and converters useful for normalizing scraped content into agent-friendly format.
markdown-it
Pluggable, fast Markdown parser for Node.js — CommonMark-compliant.
gomarkdown
Markdown parser and renderer for Go.
markitdown
Microsoft's any-file-to-Markdown converter — handles PDFs, Office docs, images (OCR), audio (transcription), HTML, and more. Canonical input pre-processor for LLMs.
Document & Component Extraction
Extracting structured data from documents (PDFs, forms, invoices) — overlaps with web scraping when targets are document-heavy.
sparrow
ML-driven data extraction from documents — invoices, receipts, forms — with LLM/VLM backends.
unstructured
Open-source library for partitioning, cleaning, and chunking unstructured documents (PDF, DOCX, HTML, images, email) for LLM ingestion.
docling
IBM's document conversion toolkit — PDF, DOCX, PPTX, images → structured Markdown/JSON with layout understanding.
pypdf
Pure-Python PDF library — read, split, merge, extract text and metadata.
PyMuPDF
High-performance PDF / e-book library (MuPDF bindings) — fast text and image extraction.
Mobile
Scraping and automation targeting mobile apps rather than the web.
mobile-mcp
MCP server for automating mobile devices (iOS / Android) — scrape, interact, extract.
Use Cases & Recipes
Real-world scraper collections — useful as references and patterns to copy.
city-scrapers
Civic-tech project scraping public meeting agendas and minutes from local government sites.
scrapfly-scrapers
Collection of production-grade scraping recipes from Scrapfly — covers many real-world targets (LinkedIn, Amazon, Zillow, etc.).
Israel-Specific
israeli-bank-scrapers
Scrapers for major Israeli banks and credit-card companies — exports transactions in a normalized format.
Awesome Lists
Upstream curated lists worth referencing.
awesome-web-scraping
Comprehensive awesome-list of web-scraping tools across languages.
awesome-mcp-api
Awesome-list of MCP servers exposing APIs and data sources to LLM agents — useful for finding scrape/fetch-style MCP tools.
Related Indexes
- AI-Browser-Tools — browser automation and browserless agentic browsing. See SCOPE.md for disambiguation.
License
