Content Core Processors

April 12, 2026 · View on GitHub

This document describes how Content Core extracts content from different source types. Processors are organized into three categories: URL, document, and media.

Overview

Content Core automatically selects the appropriate processor based on input:

  • URL provided -- routes to a URL processor (or YouTube processor for YouTube links)
  • File path provided -- file type is detected, then routed to the matching document or media processor
  • Text provided -- handled by the text processor

File type detection is done in pure Python using binary signatures and content analysis. No system library (like libmagic) is required.

URL Processors

Located in src/content_core/processors/url/.

Engine Selection

The url_engine setting controls which processor handles web URLs. When set to auto (the default), Content Core tries engines in this order:

  1. Firecrawl -- if FIRECRAWL_API_KEY is set
  2. Jina -- uses Jina Reader API (works without API key, but JINA_API_KEY avoids rate limits)
  3. Crawl4AI -- if installed (pip install content-core[crawl4ai])
  4. BeautifulSoup -- always available as the final fallback

BeautifulSoup (simple)

  • File: bs4.py
  • Fetches HTML via aiohttp and extracts meaningful text
  • No external API keys required
  • Suitable for simple pages; may struggle with JavaScript-heavy sites

Jina (jina)

  • File: jina.py
  • Uses Jina Reader API for clean content extraction
  • Works without an API key, but setting JINA_API_KEY avoids rate limits
  • Good for articles and documentation pages

Firecrawl (firecrawl)

  • File: firecrawl.py
  • Uses the Firecrawl API for high-quality web extraction
  • Requires FIRECRAWL_API_KEY
  • Supports self-hosted instances via FIRECRAWL_API_URL
  • Configurable proxy (CCORE_FIRECRAWL_PROXY, default: auto) — retries with stealth proxies on anti-bot blocks
  • Configurable wait time (CCORE_FIRECRAWL_WAIT_FOR, default: 3000ms) — waits for dynamic content before extraction

Reddit (automatic)

  • File: reddit.py
  • Automatically detects Reddit post URLs and extracts via the public .json endpoint
  • Extracts post title, body, metadata (author, score, subreddit), and full comment tree with nested replies
  • No API key or authentication needed for public posts
  • Falls back to the configured URL engine if JSON extraction fails
  • Supports www.reddit.com, old.reddit.com, and new.reddit.com

Crawl4AI (crawl4ai)

  • File: crawl4ai.py
  • Two modes: local browser automation or Docker API
  • Local mode (default): Requires pip install content-core[crawl4ai] and python -m playwright install --with-deps
  • Docker mode: Set CRAWL4AI_API_URL (e.g., http://localhost:11235) to use a remote Crawl4AI container — no local Playwright needed
  • No API keys needed; all processing happens locally or on your server
  • Handles JavaScript-heavy sites well

Document Processors

Located in src/content_core/processors/document/.

Engine Selection

The document_engine setting controls document processing. When set to auto (the default):

  1. Docling -- tried first if installed
  2. Simple -- pdfplumber for PDF, fast-ebook for EPUB, native parsers for Office formats

PDF (Simple Engine)

  • File: document/pdf.py
  • Uses pdfplumber for text and table extraction
  • Features:
    • Automatic table detection and conversion to markdown
    • Text cleaning (ligatures, whitespace, hyphenation normalization)

EPUB

  • File: document/epub.py
  • Uses fast-ebook (Rust-powered) for EPUB2/EPUB3 extraction
  • Converts EPUB content to markdown

DOCX

  • File: document/docx.py
  • Extracts text from Word documents using python-docx

PPTX

  • File: document/pptx.py
  • Extracts text from PowerPoint slides

XLSX

  • File: document/xlsx.py
  • Extracts data from Excel spreadsheets

Docling (Optional)

  • File: document/docling.py
  • Requires pip install content-core[docling]
  • Supports PDF, DOCX, PPTX, XLSX, Markdown, AsciiDoc, HTML, CSV, and images
  • Configurable output format: markdown (default), HTML, or JSON
  • Provides richer structural parsing than the simple engine

Media Processors

Located in src/content_core/processors/media/.

Audio

  • File: media/audio.py
  • Transcribes audio files using speech-to-text (default: OpenAI Whisper)
  • Supported formats: MP3, WAV, M4A, FLAC, OGG
  • Uses ffmpeg/ffprobe for duration detection and segment splitting (stream copy, no re-encoding)
  • Features:
    • Files longer than 10 minutes are automatically split into segments
    • Segments are transcribed in parallel with configurable concurrency (1-10, default 3)
    • Results are assembled in correct order regardless of completion time
    • Custom STT provider and model can be specified per call

Video

  • File: media/video.py
  • Extracts audio from video files, then transcribes using the audio processor
  • Supported formats: MP4, AVI, MOV, MKV
  • Uses ffmpeg/ffprobe for audio stream selection and extraction

YouTube Processor

  • File: processors/url/youtube.py
  • Extracts transcripts from YouTube videos
  • Uses youtube-transcript-api (primary) with pytubefix as fallback
  • Supports configurable language preferences via CCORE_YOUTUBE_LANGUAGES
  • Automatic retry with backoff for rate limiting

Text Processor

  • File: processors/text.py
  • Handles direct text input
  • Automatically detects HTML content and converts it to markdown using markdownify
  • Plain text passes through unchanged

File Type Detection

  • Located in src/content_core/content/identification/file_detector.py
  • Pure Python implementation, no system dependencies
  • Detection methods:
    • Binary signature matching for PDF, images, audio, video, and archives
    • ZIP structure inspection for Office formats (DOCX, XLSX, PPTX) and EPUB
    • Content analysis for text-based formats (HTML, XML, JSON, YAML, CSV, Markdown)
  • Reads only the first 512 bytes for binary signatures and 1024 bytes for text content analysis
  • Works regardless of file extension