Unit tests (routing/scoring helpers)

May 2, 2026 · View on GitHub

LLM Council

5 AI models debate every decision. The best answer wins.
Better than any single model. Open-source. Pluggable.

No single AI model is right about everything. Claude is careful but sometimes overly cautious. GPT-4o is creative but sometimes halluccinates. Gemini is fast but sometimes shallow. Llama is free but sometimes wrong. What if they all debated and peer-reviewed each other?

How It Works

┌─────────────────────────────────────────────────────┐
│                  YOUR QUESTION                       │
└────────────────────────┬────────────────────────────┘
                         │
         ┌───────────────┼───────────────┐
         ▼               ▼               ▼
    ┌─────────┐    ┌──────────┐    ┌──────────┐
    │  Claude  │    │  GPT-4o  │    │  Gemini  │    + Llama, Sonnet
    │  Opus    │    │          │    │  Flash   │
    └────┬────┘    └────┬─────┘    └────┬─────┘
         │              │               │
         ▼              ▼               ▼
    ┌─────────────────────────────────────────┐
    │     STAGE 1: Independent Opinions       │
    │     Each model answers separately       │
    └────────────────────┬────────────────────┘
                         │
    ┌────────────────────┴────────────────────┐
    │    STAGE 2: Anonymized Peer Review      │
    │    Each model scores the others' plans  │
    │    against a rubric (blind review)      │
    └────────────────────┬────────────────────┘
                         │
    ┌────────────────────┴────────────────────┐
    │    STAGE 3: Chairman Synthesis          │
    │    Opus merges ranked plans into one    │
    │    final answer with confidence score   │
    └────────────────────┬────────────────────┘
                         │
    ┌────────────────────┴────────────────────┐
    │           FINAL ANSWER                   │
    │    Confidence: 92% (4/5 agreement)      │
    │    Dissent: Llama disagreed on X        │
    └─────────────────────────────────────────┘

Three Modes

Mode	When	Cost	Speed	Quality
Full Council	Architecture, security, migrations	~$2-5	30-60s	Highest
Fast Council	Medium decisions, quick polls	~$0.10	5-10s	Good
Single Model	Simple tasks, bulk operations	~$0.01	1-3s	Baseline

The 5 Council Members

Member	Provider	Role	Strength
Claude Opus	Anthropic	Chairman	Careful reasoning, safety, synthesis
Claude Sonnet	Anthropic	Member	Fast + accurate, good code
GPT-4o	OpenAI	Member	Creative, broad knowledge
Gemini Flash	Google	Member	Speed, multimodal, large context
Llama 3 70B	Groq (free)	Member	Open-source perspective, zero cost

Council presets (standard, VC, humanitarian)

This repository consolidates what used to be described as separate “product flavors” into three presets that share the same five underlying models (so one .env drives every preset). Only names and role descriptions change:

Preset	Id	When to use
Standard / MAMA	`standard`	General engineering, product, and MAMA-routed work (default).
VC deal analysis	`vc`	Diligence-style framing — memos, terms, market story, data-room scale.
Humanitarian & impact	`humanitarian`	Ethics, stakeholders, field constraints, social-impact programs.

API / CLI: pass variant in JSON (POST /api/ask) or --variant=vc / --variant=humanitarian with npm run test:llms. Use GET /api/councils for labels and descriptions.

Why This Works

Research shows multi-model deliberation outperforms any single model:

Blind spots eliminated — what Claude misses, GPT-4o catches (and vice versa)
Hallucination reduced — if 4/5 models agree, the answer is more likely correct
Bias diversified — each model was trained differently, on different data
Confidence calibrated — unanimous agreement = high confidence, split vote = flag for human
Cost optimized — simple tasks use the cheapest model, complex tasks justify the full council

Auto-Routing by Complexity

The system scores task complexity (1-10) and routes automatically:

Score 1-3 → Single Model (Groq Llama, free)
Score 4-6 → Fast Council (all 5 vote, majority wins)
Score 7-10 → Full Council (3-stage deliberation)

Scoring factors: keywords (architecture=8, migration=9, UI=3), file count, priority label, department.

Confidence & Consensus

Agreement	Confidence	Action
5/5 agree	95%+	Auto-execute
4/5 agree	80%	Execute + log dissent
3/5 agree	60%	Flag for human review
No majority	<50%	Escalate, do not execute

Dissent is valuable. When one model disagrees, we log why. Sometimes the dissenter is right.

Rubric System

Default scoring categories (each 0-10):

Category	What It Measures
Correctness	Is the answer factually right?
Completeness	Does it cover all aspects?
Security	Any vulnerabilities introduced?
Performance	Is it efficient?
Maintainability	Can someone else understand it?

Per-product rubric overrides available (e.g., trading accuracy matters more for Grail, UX matters more for LT).

MAMA Integration

In MAMA, the council is available via:

/council [task]           → auto-routed by complexity
/council fast [question]  → force fast mode
/council full [task]      → force full deliberation
/council status           → recent sessions

API:

POST /api/council/run     → { task, mode?, product? }
GET  /api/council/sessions → list recent sessions

Improvements Over Karpathy's Original

Feature	Karpathy's llm-council	Our LLM Council
Modes	Single mode	3 modes (full/fast/single)
Auto-routing	Manual	Complexity-scored auto-routing
Confidence	None	Quantified consensus scoring
Persistence	None	Supabase session logging
Integration	Standalone	MAMA Slack + API + autonomy engine
Cost control	None	Daily caps, caching, budget governor
Rubrics	Fixed	Per-product customizable
Dissent tracking	None	Logged with reasoning

Use Cases Beyond Code

VC Deal Analysis — 5 models each grade a deal, peer-review, synthesize (used on WHOOP + Dyme)
Content Moderation — multi-model consensus on whether content is appropriate
Medical Triage — cross-reference symptom analysis across models (with human oversight)
Legal Review — multiple perspectives on contract clauses
Architecture Decisions — debate tradeoffs before committing

Quick Start

git clone https://github.com/OliWoods-Org/llm-council.git
cd llm-council
npm install && npm run build

Copy .env.example to .env and set provider keys (ANTHROPIC_API_KEY, OPENAI_API_KEY, GOOGLE_API_KEY or GEMINI_API_KEY, GROQ_API_KEY).

# Local UI + API: POST /api/ask, GET /api/health
npm run dev:server
# Open http://127.0.0.1:3847/

# CLI: all models on the standard council (default preset)
npm run test:llms -- "Your question here"

# VC or humanitarian preset (same APIs, different member labels)
npm run test:llms -- --variant=vc "Summarize key risks for this seed round"
npm run test:llms -- --variant=humanitarian "Prioritize aid given constrained budget"

# Unit tests (routing/scoring helpers)
npm test

See IMPROVEMENTS.md for roadmap and MAMA integration notes.

Or use via MAMA: /council Design a database schema for a marketplace

Contributing

We need developers, ML researchers, and people who care about AI decision quality. PRs welcome.

Project	Description
Humanitarian preset (`humanitarian`)	In-repo — use council variant `humanitarian` (see above)
MAMA	AI Chief of Staff — 85+ agent teams
Karpathy's llm-council	Original inspiration

An OliWoods Foundation Project
Better decisions through multi-model deliberation

_{Built with MAMA · Powered by CoFounder

"The test of a first-rate intelligence is the ability to hold two opposing ideas in mind at the same time and still retain the ability to function."

— F. Scott Fitzgerald

Apache 2.0 — Fork it. Debate it. Build better AI.}

LLM Council

5 AI models debate every decision. The best answer wins.Better than any single model. Open-source. Pluggable.

5 AI models debate every decision. The best answer wins.
Better than any single model. Open-source. Pluggable.