Integration Analysis: kreuzberg-dev/html-to-markdown

May 30, 2026 · View on GitHub

Overview

This document analyzes the best experiences from kreuzberg-dev/html-to-markdown (v3.5.x) and how they have been integrated into the web-capture project.

Key Features of html-to-markdown

Feature	Description	Integration Value
High Performance	150-280 MB/s throughput, Rust-powered core	High - adds a faster converter option for JS and Rust
Structured Results	`ConversionResult` with content, metadata, tables, images, warnings	High - enriches our API responses
Metadata Extraction	Title, links, headings, images, JSON-LD, Microdata, RDFa, Open Graph	High - replaces custom metadata logic
Table Extraction	Structured cell data with headers, alignment, rendered markdown	Medium - enhances table handling
Visitor Pattern	Custom callbacks for content filtering, URL rewriting	Medium - enables extensibility
HTML Sanitization	Built-in sanitization via ammonia	Medium - replaces manual cleaning
Multiple Output Formats	Markdown, Djot, Plain Text	Low - we primarily need Markdown
12 Language Bindings	Consistent output across Rust, Node.js, Python, etc.	High - both our JS and Rust use same core
CommonMark Compliance	Standards-based markdown output	Medium - improves output quality

What We Integrated

1. Node.js: `@kreuzberg/html-to-markdown-node` (v3.5.x)

Package: @kreuzberg/html-to-markdown-node

Added as an optional, high-performance converter that can be selected via configuration or query parameter. The existing Turndown-based converter remains as the default for backward compatibility.

Benefits:

10-80x faster conversion than Turndown
Structured results with metadata, tables, images
Built-in HTML sanitization
CommonMark compliant output

2. Rust: `html-to-markdown-rs` (v3.5.x)

Crate: html-to-markdown-rs

Adds html-to-markdown-rs as an alternate high-performance converter while keeping the existing html2md path as the Rust default for backward compatibility.

Benefits:

Same Rust core as the Node.js binding (consistent output)
Structured conversion results
Built-in metadata extraction
Better table handling

3. Structured Conversion Results

Both implementations now return structured results including:

content: The converted markdown
metadata: Extracted page metadata (title, description, links, headings, images)
tables: Structured table data extracted during conversion
images: Inline image extraction results when available
warnings: Any non-fatal processing warnings

4. Enhanced Metadata Extraction

The html-to-markdown library extracts richer metadata than our custom implementation:

Open Graph tags (og:title, og:description, og:image)
Twitter Card metadata
JSON-LD structured data
Microdata (itemscope, itemtype, itemprop)
RDFa markup
Link classification (internal, external, anchor, email, phone)

What We Kept

Custom LaTeX extraction: html-to-markdown doesn't handle LaTeX formula extraction from Habr, KaTeX, or MathJax - our custom implementation remains
Custom post-processing: Unicode normalization, LaTeX spacing, bold formatting fixes remain for the Turndown path
URL absolutification: Our runtime JS hook for dynamic URLs is unique to web-capture
Browser automation: The fetching and rendering layer is independent of conversion

API Changes

Query Parameter: `converter`

The /markdown endpoint now accepts a converter query parameter:

converter=turndown (JavaScript default) - Use existing Turndown-based conversion
converter=html2md (Rust default) - Use existing html2md-based conversion
converter=kreuzberg - Use html-to-markdown for high-performance conversion with structured results

Response Format

When using the kreuzberg converter, the /markdown endpoint can optionally return JSON with structured results:

GET /markdown?url=https://example.com&converter=kreuzberg&format=json

{
  "content": "# Example\n\nThis is the page content...",
  "metadata": {
    "title": "Example Domain",
    "links": [...],
    "headings": [...],
    "images": [...]
  },
  "tables": [...],
  "images": [...],
  "warnings": []
}

Performance Comparison

Metric	Turndown (JS)	html2md (Rust)	html-to-markdown
Throughput	~5-10 MB/s	~20-40 MB/s	150-280 MB/s
Structured results	No	No	Yes
Metadata extraction	Custom	None	Built-in
Table extraction	GFM plugin	Basic	Structured
Sanitization	Manual (Cheerio)	Manual (scraper)	Built-in (ammonia)
CommonMark	Partial	Partial	Full