Using the Content Core Library
April 12, 2026 · View on GitHub
This guide covers the Python API, configuration, and usage patterns for Content Core v2.0.
Basic Usage
Extraction
The primary function is extract_content, which accepts keyword arguments and returns an ExtractionOutput:
import content_core
# Extract from a URL
result = await content_core.extract_content(url="https://example.com")
print(result.content)
# Extract from a file
result = await content_core.extract_content(file_path="document.pdf")
print(result.content)
# Extract from raw text
result = await content_core.extract_content(content="Some text content.")
print(result.content)
Summarization
import content_core
summary = await content_core.summarize("Long article text here...", context="bullet points")
print(summary)
# Other context examples
summary = await content_core.summarize(text, context="explain to a child")
summary = await content_core.summarize(text, context="executive summary")
summary = await content_core.summarize(text, context="action items")
Extraction by Source Type
URLs
# Auto-detect best engine
result = await content_core.extract_content(url="https://example.com")
# Force a specific engine
from content_core import ContentCoreConfig
config = ContentCoreConfig(url_engine="firecrawl")
result = await content_core.extract_content(url="https://example.com", config=config)
Documents
# PDF
result = await content_core.extract_content(file_path="report.pdf")
# Word document
result = await content_core.extract_content(file_path="document.docx")
# PowerPoint
result = await content_core.extract_content(file_path="slides.pptx")
# Excel
result = await content_core.extract_content(file_path="data.xlsx")
# Use Docling engine for richer parsing
from content_core import ContentCoreConfig
config = ContentCoreConfig(document_engine="docling", docling_output_format="html")
result = await content_core.extract_content(file_path="report.pdf", config=config)
Audio and Video
# Audio transcription
result = await content_core.extract_content(file_path="interview.mp3")
# Video (audio is extracted and transcribed automatically)
result = await content_core.extract_content(file_path="lecture.mp4")
# With custom speech-to-text model
from content_core import ContentCoreConfig
config = ContentCoreConfig(audio_provider="openai", audio_model="whisper-1")
result = await content_core.extract_content(file_path="interview.mp3", config=config)
YouTube
result = await content_core.extract_content(url="https://www.youtube.com/watch?v=dQw4w9WgXcQ")
print(result.content) # Video transcript
Text with HTML Auto-Detection
# Plain text passes through unchanged
result = await content_core.extract_content(content="Plain text here.")
# HTML content is automatically detected and converted to markdown
result = await content_core.extract_content(
content="<h1>Title</h1><p>Paragraph with <a href='url'>link</a></p>"
)
Configuration
Content Core uses ContentCoreConfig backed by pydantic-settings. Configuration is resolved in priority order:
- Constructor args (Python API) or command flags (CLI) — highest priority
- Environment variables (
CCORE_*prefix) - Config file (
~/.content-core/config.toml) - Defaults — lowest priority
Config File
Set persistent defaults via CLI or edit the file directly:
content-core config set llm_provider anthropic
content-core config set llm_model claude-sonnet-4-20250514
content-core config set url_engine firecrawl
Or edit ~/.content-core/config.toml:
llm_provider = "anthropic"
llm_model = "claude-sonnet-4-20250514"
url_engine = "firecrawl"
youtube_languages = ["en", "pt"]
In Code
from content_core import ContentCoreConfig
config = ContentCoreConfig(
url_engine="firecrawl",
document_engine="docling",
audio_concurrency=5,
llm_provider="openai",
llm_model="gpt-4o-mini",
stt_provider="openai",
stt_model="whisper-1",
stt_timeout=3600,
youtube_languages=["en", "pt"],
)
result = await content_core.extract_content(url="https://example.com", config=config)
Via Environment Variables
All config fields use the CCORE_ prefix:
CCORE_URL_ENGINE=firecrawl
CCORE_DOCUMENT_ENGINE=auto
CCORE_AUDIO_CONCURRENCY=5
CCORE_LLM_PROVIDER=openai
CCORE_LLM_MODEL=gpt-4o-mini
CCORE_STT_PROVIDER=openai
CCORE_STT_MODEL=whisper-1
CCORE_STT_TIMEOUT=3600
CCORE_YOUTUBE_LANGUAGES=en,pt
CCORE_FIRECRAWL_PROXY=auto
CCORE_FIRECRAWL_WAIT_FOR=3000
FIRECRAWL_API_URL=http://localhost:3002
CRAWL4AI_API_URL=http://localhost:11235
API keys for external services use their standard variable names:
OPENAI_API_KEY=sk-...
FIRECRAWL_API_KEY=fc-...
JINA_API_KEY=jina-...
Engine Selection
URL Engines
The url_engine setting controls how web pages are extracted:
| Engine | Description | Requirements |
|---|---|---|
auto (default) | Tries engines in order: Firecrawl, Jina, Crawl4AI, BeautifulSoup | Depends on available API keys |
simple | BeautifulSoup-based extraction | None |
firecrawl | Firecrawl API | FIRECRAWL_API_KEY |
jina | Jina Reader API | Optional JINA_API_KEY |
crawl4ai | Local browser-based extraction | pip install content-core[crawl4ai] + Playwright |
For self-hosted Firecrawl, set FIRECRAWL_API_URL to your instance URL.
Document Engines
The document_engine setting controls how files (PDF, DOCX, PPTX, XLSX) are processed:
| Engine | Description | Requirements |
|---|---|---|
auto (default) | Tries Docling first, falls back to simple | Depends on installed extras |
simple | pdfplumber for PDF, fast-ebook for EPUB, python-docx/openpyxl/python-pptx for Office | None (included) |
docling | Docling library for rich document parsing | pip install content-core[docling] |
AI Providers
Content Core uses Esperanto as a unified abstraction layer for LLM and Speech-to-Text providers. This means you can use any provider supported by Esperanto without changing your code — just update the config.
Supported Providers
| Provider | LLM | STT | API Key |
|---|---|---|---|
openai (default) | GPT-4o, GPT-4o-mini, etc. | Whisper | OPENAI_API_KEY |
anthropic | Claude Sonnet, Claude Opus, etc. | — | ANTHROPIC_API_KEY |
google | Gemini models | — | GOOGLE_API_KEY |
groq | LLaMA, Mixtral, etc. | Whisper | GROQ_API_KEY |
deepseek | DeepSeek models | — | DEEPSEEK_API_KEY |
ollama | Local models | — | — (local) |
openrouter | Multiple providers | — | OPENROUTER_API_KEY |
For a complete and up-to-date list of providers and models, see the Esperanto documentation.
LLM Configuration (Summarization)
# Via config file (persistent)
content-core config set llm_provider anthropic
content-core config set llm_model claude-sonnet-4-20250514
# Via environment variables (per-session)
export CCORE_LLM_PROVIDER=anthropic
export CCORE_LLM_MODEL=claude-sonnet-4-20250514
# Via Python API (per-call)
config = ContentCoreConfig(llm_provider="anthropic", llm_model="claude-sonnet-4-20250514")
summary = await content_core.summarize("text", "bullet points", config=config)
Use summary_model to set a different model specifically for summarization while keeping llm_model as the default for other LLM operations.
STT Configuration (Audio/Video Transcription)
# Via config file (persistent)
content-core config set stt_provider openai
content-core config set stt_model whisper-1
# Via environment variables (per-session)
export CCORE_STT_PROVIDER=openai
export CCORE_STT_MODEL=whisper-1
# Via Python API (per-call)
config = ContentCoreConfig(stt_provider="openai", stt_model="whisper-1")
result = await content_core.extract_content(file_path="audio.mp3", config=config)
Use audio_provider and audio_model together to override the STT provider for a specific call without changing the default.
Audio Processing
Long audio files are automatically split into segments and transcribed in parallel.
Concurrency
Control the number of simultaneous transcriptions (1-10, default 3):
CCORE_AUDIO_CONCURRENCY=5
Or in code:
config = ContentCoreConfig(audio_concurrency=5)
Higher values speed up processing of long files but may hit API rate limits.
Custom STT Models
Override the speech-to-text provider and model per call:
config = ContentCoreConfig(audio_provider="openai", audio_model="whisper-1")
result = await content_core.extract_content(file_path="audio.mp3", config=config)
Both audio_provider and audio_model should be specified together for a complete override. If only audio_model is provided, it is used with the default provider.
Docling Integration
Docling provides advanced document parsing for PDF, DOCX, PPTX, XLSX, Markdown, AsciiDoc, HTML, CSV, and images.
Installation
pip install content-core[docling]
Usage
# Set globally via config
config = ContentCoreConfig(document_engine="docling")
# Or per-call
config = ContentCoreConfig(document_engine="docling", docling_output_format="html")
result = await content_core.extract_content(file_path="report.pdf", config=config)
Enrichment Features
Docling supports optional enrichment features that require additional models. These are disabled by default to avoid extra download and processing time.
| Flag | What it does | Default | Impact |
|---|---|---|---|
docling_ocr | OCR for scanned PDFs | true | Medium — runs per page |
docling_formulas | Extract equations as LaTeX | false | High — downloads ~500MB model, CPU/CUDA only |
docling_vision | Image descriptions + chart data extraction | false | High — downloads ~2GB model |
Via config file (persistent)
content-core config set docling_formulas true
content-core config set docling_vision true
Via CLI (per-call)
content-core extract --engine docling --formulas paper.pdf
content-core extract --engine docling --pictures --no-ocr paper.pdf
Via Python API
from content_core import ContentCoreConfig
config = ContentCoreConfig(
document_engine="docling",
docling_formulas=True,
docling_vision=True,
)
result = await content_core.extract_content(file_path="paper.pdf", config=config)
Via MCP
The extract_content MCP tool accepts formulas, pictures, and no_ocr parameters:
extract_content(file_path="paper.pdf", engine="docling", formulas=true, pictures=true)
Note: Enrichment flags are only applied when
document_engine="docling". A warning is logged if flags are set without the Docling engine.
Proxy Configuration
Content Core uses standard HTTP proxy environment variables:
HTTP_PROXY=http://proxy.example.com:8080
HTTPS_PROXY=http://proxy.example.com:8080
NO_PROXY=localhost,127.0.0.1,internal.example.com
# With authentication
HTTP_PROXY=http://user:password@proxy.example.com:8080
All network requests (aiohttp, YouTube fetching, Crawl4AI, Esperanto models) automatically use these variables. No additional Content Core configuration is needed.
Note: Firecrawl does not support client-side proxy configuration. Configure proxy on the Firecrawl server side instead.
Custom Prompt Templates
Content Core uses built-in prompts for summarization. You can override them by setting the PROMPT_PATH environment variable:
PROMPT_PATH=/path/to/your/custom/prompts
When a prompt template is requested, Content Core checks your custom directory first. If the template is not found there, it falls back to the built-in prompts.
Retry Behavior
Content Core automatically retries transient failures (network timeouts, API rate limits, connection errors) with exponential backoff and jitter. This applies to all external operations including URL extraction, audio transcription, YouTube fetching, and LLM calls.
Retries are configured with sensible defaults and require no setup. If you need to tune retry behavior for specific scenarios (e.g., aggressive rate limits), you can override settings via environment variables:
# Pattern: CCORE_{OPERATION}_{PARAM}
CCORE_YOUTUBE_MAX_RETRIES=10
CCORE_URL_API_MAX_RETRIES=5
CCORE_AUDIO_MAX_RETRIES=5
File Type Detection
Content Core uses a pure Python file type detector based on binary signatures and content analysis. No system dependencies (like libmagic) are required. Detection works regardless of file extension and supports 25+ formats across documents, images, media, text, and archives.