dspy-plasmate

DSPy integration for Plasmate, the browser engine for AI agents.

Plasmate converts HTML to a Semantic Object Model (SOM), enabling web-augmented RAG and agent tools with 16x fewer tokens compared to raw HTML.

Installation

pip install dspy-plasmate

You'll also need Plasmate installed:

# macOS
brew install nickshanks/tap/plasmate

# Or build from source
cargo install plasmate

Quick Start

Fetch a Web Page

from dspy_plasmate import plasmate_fetch

# Get readable text from any URL
content = plasmate_fetch("https://example.com")
print(content)

Use as a DSPy Tool

import dspy
from dspy_plasmate import PlasmateFetchTool

# Configure DSPy with your LLM
dspy.configure(lm=dspy.LM("openai/gpt-4o-mini"))

# Create a ReAct agent with web browsing capability
tool = PlasmateFetchTool()

react = dspy.ReAct(
    signature="question -> answer",
    tools=[tool],
)

result = react(question="What is the main feature of Plasmate according to plasmate.app?")
print(result.answer)

Web-Augmented RAG

from dspy_plasmate import PlasmateRetriever, WebSearchModule

# Retrieve content from multiple URLs
retriever = PlasmateRetriever(k=5)
result = retriever(
    query="What is machine learning?",
    urls=[
        "https://en.wikipedia.org/wiki/Machine_learning",
        "https://en.wikipedia.org/wiki/Deep_learning",
    ]
)

for passage in result.passages:
    print(passage.long_text[:200])

# Or use the pre-built search module
searcher = WebSearchModule(urls=[
    "https://en.wikipedia.org/wiki/Python_(programming_language)",
])
answer = searcher(question="Who created Python?")
print(answer.answer)

Question Answering

from dspy_plasmate import WebQAModule

qa = WebQAModule()
result = qa(
    url="https://en.wikipedia.org/wiki/Rust_(programming_language)",
    question="What makes Rust memory-safe?"
)
print(result.answer)

Summarization

from dspy_plasmate import WebSummarizeModule

summarizer = WebSummarizeModule()
result = summarizer(url="https://news.ycombinator.com")
print(result.summary)

Components

PlasmateRetriever

A DSPy retriever that fetches web pages and returns them as passages.

from dspy_plasmate import PlasmateRetriever

retriever = PlasmateRetriever(
    k=3,                    # Number of passages per URL
    text_only=True,         # Extract readable text (vs. full SOM JSON)
    timeout=30,             # Request timeout in seconds
    headers={"Authorization": "Bearer ..."},  # Optional headers
)

result = retriever(urls=["https://example.com"])

PlasmateFetchTool

A tool for DSPy agents that fetches web pages.

from dspy_plasmate import PlasmateFetchTool

tool = PlasmateFetchTool(
    text_only=True,         # Readable text output
    timeout=30,
)

# Use directly
content = tool("https://example.com")

# Or in a ReAct agent
react = dspy.ReAct(signature="question -> answer", tools=[tool])

Signatures

Pre-defined DSPy signatures for common web tasks:

• WebSearch - RAG-style Q&A with web context
• WebSummarize - Summarize a web page
• WebQA - Answer questions about a specific page
• WebExtract - Extract structured data from pages
• WebCompare - Compare content across pages

Modules

Pre-built modules combining retrieval and generation:

• WebSearchModule - Answer questions using multiple URLs
• WebSummarizeModule - Summarize any URL
• WebQAModule - Q&A about a specific page

Examples

See the examples/ directory for complete working examples:

python examples/web_qa.py

Why Plasmate?

Traditional web scraping sends raw HTML to LLMs, wasting tokens on markup, scripts, and styling. Plasmate extracts a Semantic Object Model that preserves structure while removing noise:

Approach	Tokens (typical page)
Raw HTML	~50,000
Plasmate SOM	~3,000
Plasmate Text	~1,500

This means:

• Lower costs - 16x fewer tokens per page
• Faster responses - Less data to process
• Better context - More pages fit in context window
• Cleaner data - No script/style noise

Configuration

Custom Plasmate Path

from dspy_plasmate import PlasmateFetchTool

tool = PlasmateFetchTool(plasmate_path="/usr/local/bin/plasmate")

Custom Headers

tool = PlasmateFetchTool(headers={
    "Authorization": "Bearer your-token",
    "User-Agent": "MyBot/1.0",
})

Get Full SOM JSON

from dspy_plasmate import PlasmateFetchTool
import json

tool = PlasmateFetchTool(text_only=False)
som_json = tool("https://example.com")
data = json.loads(som_json)

License

MIT