🔮 Coven

On-Premise/Offline Multi-LLM orchestration interface — a "Council of Models" architecture where multiple LLMs deliberate on your query and a Chairman model synthesizes the best response, fully customizable and extensible.

more images below!

✨ Features

• 🤖 Council of Models — Queries multiple LLMs in parallel (e.g., Llama 3, Mistral, Gemma, DeepSeek R1, etc.) to get diverse perspectives. Basically everything that is supported by Ollama offline can be used in Coven. Coven supports also the use of multiple Ollama instances or remote ones if you prefer.
• 👑 Chairman Synthesis — A "superior" model aggregates all drafts into a single, high-quality response. The Chairman is also customizable and can be a different model from the council members.
• 🧠 Thinking Mode Support — Full support for reasoning models (e.g., DeepSeek R1) with visualized thought traces in the "X-Ray" view.
• 👁️ Multimodal Vision — Upload images (PNG, JPG, WEBP) for analysis by vision-capable models (e.g., Llama 3.2 Vision). The UI automatically displays model capability badges (🖼️ Vision, 🧠 Thinking, 🧰 Tools).
• 💾 Persistent Memory & Export — Chats are saved to disk (on linux and mac ~/.coven/chats/ and on windows C:\Users\<username>\.coven\chats\) and reloaded and restored on launch. Conversations can also be effortlessly exported to Markdown at any time.
• 📝 Automatic Summarization — Long conversations are compressed into concise summaries to maintain context without token overflow.
• 🔎 Tool Calling — Models can use tools like Web Search (DuckDuckGo) and current Time retrieval if enabled in the coven.toml configuration file.
• 📚 Built-in RAG (Retrieval-Augmented Generation) — Attach code, text, and PDFs. The application automatically chunks, embeds, and stores documents inside a local Milvus Lite vector database for each chat. Only the most mathematically relevant context is injected into the prompt, avoiding token limit overflow.
• 📊 X-Ray UI — A dedicated "Council Deliberation" expander lets you inspect raw drafts, peer critique revisions, retrieved context snippets from the vector store, and exact execution times and token counts per model.
• 🛡️ Robust Health Checks — On startup, Coven verifies all configured Ollama URLs (even across multiple instances) and checks if the requested models are pulled, alerting you to unreachable hosts or missing models so you never infer blindly.
• ⚡ Async & Streaming — Fully asynchronous pipeline with real-time streaming of the final verdict.
• ⚙️ Configurable — Easy TOML configuration for models, parameters, pipelines, and tools.

graph TD
    User([User Query]) --> FileIngest[File & History Ingestion]
    FileIngest -->|Chunk & Embed| Milvus[(Milvus Vector DB)]
    
    Milvus -->|Semantic Retrieval| Context{RAG Context}
    User --> Context
    
    Context -->|Summarization| Memory[Short-term Memory]
    Context -->|Tool Execution| Tools[Web Search / Time]
    
    subgraph Council [The Council]
        M1[Model A]
        M2[Model B]
        M3[Model C]
    end
    
    Memory --> M1
    Tools --> M1
    Memory --> M2
    Memory --> M3
    
    M1 --> D1[Draft A]
    M2 --> D2[Draft B]
    M3 --> D3[Draft C]
    
    D1 & D2 & D3 --> Critique{Critique Phase?}
    Critique -->|Yes| Review[Peer Review & Revise]
    Review --> FinalDrafts[Revised Drafts]
    Critique -->|No| FinalDrafts[Drafts]
    
    FinalDrafts --> Chairman[👑 Chairman]
    Chairman -->|Stream| Response([Final Response])

1. Context Loading & RAG

   • Memory: The system loads the last N turns of conversation. If the history is too long, it is automatically summarized by the Chairman to preserve token space.
   • Documents: Attached files (PDF, code, text) are parsed and indexed into a local Milvus vector database alongside chat history. The query triggers a semantic search, and only the most relevant snippets are passed to the models.

2. The Inquiry

   • The query is broadcast to all Council Members in parallel.
   • Tool Usage: If enabled, models can autonomously use tools like web_search or get_current_date_time to fact-check or retrieve real-time data before answering.
   • Failure Tolerance: If a model crashes or doesn't support tools, it gracefully falls back to standard text generation or abstains without halting the pipeline.

3. The Critique (Optional)

   • If enable_critique = true, members review each other's drafts.
   • Each model sees the query + its own draft + other drafts, and generates a revised version of its answer.

4. The Decree

   • The Chairman reads all (revised) drafts.
   • It synthesizes a final, authoritative response, resolving conflicts and merging the best insights from each member.
   • The result is streamed to the UI in real-time.

Prerequisites

• Python 3.13+
• Ollama running locally or on a remote host
• Pull the models you want to use:

ollama pull llama3
ollama pull mistral
ollama pull gemma:2b
...

Quick Start

# Clone and enter the project
cd coven

# Install dependencies (requires uv)
uv sync --group dev

# Launch the app
.venv/bin/streamlit run src/coven/app.py

The UI opens at http://localhost:8501. A sidebar shows model availability status.

Configuration

All settings live in coven.toml at the project root:

[ollama]
default_url = "http://localhost:11434"

[chairman]
model = "llama3"
temperature = 0.5
# ollama_url = "http://gpu-server:11434"   # optional override

[[members]]
model = "llama3"
temperature = 0.7
# ollama_url = "http://other-host:11434"   # optional: override default

[[members]]
model = "mistral"
temperature = 0.7
# ollama_url = "http://other-host:11434"   # optional: override default

[[members]]
model = "gemma:2b"
temperature = 0.7
# ollama_url = "http://other-host:11434"   # optional override

[pipeline]
enable_critique = true      # Models review each other's drafts before the final verdict
memory_turns = 5            # Number of recent turns to keep in full context

[tools]
enable_web_search = true    # Enable DuckDuckGo search for models

Multi-Ollama Support & Health Checks

Each member or the chairman can point to a different Ollama instance via the optional ollama_url field. If omitted, it falls back to [ollama].default_url. This lets you spread models across multiple endpoints: your local computer, rented GPU servers, etc.

Coven rigorously checks these connections on startup. If an endpoint is unreachable, or if a model is missing on that specific instance, the UI will warn you precisely about which models and which URLs are experiencing issues.

Prompt Templates

Prompts live as plain text files in the prompts/ directory and can be edited without touching Python:

File	Purpose
prompts/member_system.txt	Member system instruction
prompts/member_human.txt	Member query template ({query} + {history})
prompts/chairman_system.txt	Chairman system instruction
prompts/chairman_human.txt	Chairman synthesis template ({query} + {drafts})
prompts/critique.txt	Template for peer review round
prompts/summary_system.txt	Summarizer system instruction
prompts/summary_human.txt	Summarization prompt ({history})

Project Structure

coven/
├── coven.toml              # Configuration
├── prompts/                # Editable prompt templates
│   ├── member_system.txt
│   ├── member_human.txt
│   ├── chairman_system.txt
│   ├── chairman_human.txt
│   ├── critique.txt
│   ├── summary_system.txt
│   └── summary_human.txt
├── src/coven/
│   ├── config.py           # Settings dataclass + TOML loader
│   ├── models.py           # Ollama health checks + LLM factory
│   ├── prompts.py          # Loads templates from prompts/
│   ├── orchestrator.py     # Async council + chairman synthesis
│   ├── rag.py              # LangChain Milvus semantic search integration
│   ├── storage.py          # Chat persistence (JSON/SQLite)
│   ├── summarizer.py       # History summarization logic
│   ├── tools.py            # LangChain tool definitions
│   └── app.py              # Streamlit UI
├── tests/                  # Pytest unit & integration suites
│   ├── conftest.py
│   ├── test_config.py
│   ├── test_models.py
│   ├── test_orchestrator.py
│   ├── test_prompts.py
│   ├── test_rag.py
│   ├── test_summarizer.py
│   ├── test_tools.py
│   └── test_tools_integration.py
└── pyproject.toml

Development

# Run tests
uv run pytest -v

# Lint
uv run ruff check src/ tests/

# Format
uv run ruff format src/ tests/

Tech Stack

Layer	Technology
Runtime	Python 3.13
LLM Backend	Ollama
Orchestration	LangChain (LCEL)
Frontend	Streamlit
Concurrency	asyncio
Package Manager	uv

Images

License

Private — all rights reserved.