🧵 NEEDLE
August 28, 2026 · View on GitHub
🧵 NEEDLE
Navigates Every Enqueued Deliverable, Logs Effort
Deterministic bead processing with explicit outcome paths.
NEEDLE is a universal wrapper for headless coding CLI agents. It processes a shared bead queue in deterministic order, dispatching work to any headless CLI (Claude Code, OpenCode, Codex, Aider) and handling every outcome through an explicit, predefined path.
🚀 Quickstart
# Install the latest release (prebuilt binary, Linux x86_64/aarch64)
curl -fsSL https://github.com/jedarden/NEEDLE/releases/latest/download/install.sh | bash
# Or build from source (needs Rust 1.75+, see rust-toolchain.toml)
cargo install --git https://github.com/jedarden/NEEDLE
# Install a bead backend (required)
# The bead-rs backend manages your workspace's bead store (SQLite + checkpoint)
cargo install --git https://github.com/jedarden/bead-rs --bin bead
# See https://github.com/jedarden/bead-rs for backend details and prebuilt installers (coming soon)
# Configure your bead workspace backend (required)
cd /path/to/your/workspace
cat > .needle.yaml << 'EOF'
bead_cli:
backend: bead-rs # or 'bead-forge' for legacy bf/br workspaces
EOF
# This file tells needle which bead CLI backend to use
# Initialise the workspace's bead store
bead init --prefix <short-name>
# Verify everything resolves
needle doctor
# Run a worker
needle run --agent claude --identity alpha
The installer drops needle in ~/.local/bin (override with NEEDLE_INSTALL_PATH);
make sure that directory is on your PATH.
🔒 Security Note
The installer automatically verifies SHA-256 checksums to ensure the downloaded binary has not been corrupted or tampered with. This verification is enabled by default for your protection.
Checksum verification safeguards against:
- Corrupted downloads that could crash or behave unpredictably
- Tampered binaries that could execute arbitrary malicious code
- Supply chain attacks where a malicious actor modifies releases
To see full security options and tradeoffs:
curl -fsSL https://github.com/jedarden/NEEDLE/releases/latest/download/install.sh | bash -s -- --help
⚠️ Warning: The installer supports an opt-out flag (
--skip-checksum) for emergency recovery scenarios, but this is strongly discouraged except as a temporary workaround when checksums.txt is unavailable due to network/infrastructure issues. See the help output for full security details.
A worker starts, claims the next bead, dispatches to your chosen agent CLI, and loops. Multiple workers can run in parallel against the same workspace — coordination is handled by the shared bead queue (no central orchestrator).
See docs/examples/ for end-to-end configurations.
🧶 What is a bead?
A bead is a work item — the unit of work NEEDLE processes. Think of it as a structured task ticket: a title, a body describing the deliverable and acceptance criteria, a status (open, in_progress, done), and optional metadata like priority and dependencies.
Beads live in a bead store — a pluggable backend managed by a bead CLI. The current primary backend is bead-rs, which uses a SQLite database (beads.db) plus a checkpoint directory (.beads/checkpoint/ with current.json, forensic.jsonl, and objects/). The legacy bead-forge backend uses issues.jsonl.
# Create a bead (bead-rs backend)
bead create --title "Add pagination to search results" --priority 2 --issue-type task
# List open beads
bead list --status open
# NEEDLE does the rest: claims, dispatches, and closes beads automatically
NEEDLE workers read from this store, claim the next available bead atomically, dispatch it to your chosen agent CLI, and close it on success — then loop.
🤔 Why NEEDLE
Existing agent orchestration tools are built for one of two shapes:
- Conversational frameworks (LangGraph, AutoGen, CrewAI) assume a chat loop with a human-in-the-loop or another LLM. They are bad at headless, long-running, cost-bounded work.
- Workflow engines (Temporal, Argo Workflows, Inngest) assume each step is deterministic code. They are bad at non-deterministic agent steps whose outcomes have to be classified and routed.
NEEDLE is the missing middle: a deterministic state machine that drives non-deterministic agents. Every outcome an agent can produce has an explicit handler. The agent's work is fuzzy; the orchestration around it is not.
🧠 Core Principle
NEEDLE is a state machine, not a script. Every bead transitions through a finite set of states, and every transition has a defined handler. There are no implicit fallbacks, no swallowed errors, no undefined paths.
If an outcome can happen, it has a handler.
If it doesn't have a handler, it cannot happen.
🔄 The NEEDLE Algorithm
A single worker executes this loop indefinitely:
┌─────────────────────────────────────────────────────┐
│ │
│ ┌───────────┐ │
│ │ 🔍 SELECT │◄────────────────────────────────┐ │
│ └─────┬─────┘ │ │
│ │ │ │
│ ▼ │ │
│ ┌───────────┐ race lost ┌──────────┐ │ │
│ │ 🔒 CLAIM │──────────────►│ 🔁 RETRY │─────┘ │
│ └─────┬─────┘ └──────────┘ │
│ │ claimed │
│ ▼ │
│ ┌───────────┐ │
│ │ 📋 BUILD │ │
│ └─────┬─────┘ │
│ │ │
│ ▼ │
│ ┌───────────┐ │
│ │ 🚀 DISPATCH│ │
│ └─────┬─────┘ │
│ │ │
│ ▼ │
│ ┌───────────┐ │
│ │ ⏳ EXECUTE │ │
│ └─────┬─────┘ │
│ │ │
│ ▼ │
│ ┌───────────┐ │
│ │ 📊 OUTCOME │ │
│ └─────┬─────┘ │
│ │ │
│ ├── ✅ success ──► close bead ──────────────┘
│ ├── ❌ failure ──► log + release ────────────┘
│ ├── ⏰ timeout ──► release + defer ──────────┘
│ └── 💀 crash ────► release + alert ──────────┘
│ │
└─────────────────────────────────────────────────────┘
📐 Algorithm Steps
🔍 Step 1: Select
Query the bead queue for the next claimable bead in deterministic priority order. Selection is not random — given the same queue state, every worker computes the same ordering. Ties are broken by creation time (oldest first).
🔒 Step 2: Claim
Attempt an atomic claim via the bead CLI (bead claim for bead-rs, bf claim for bead-forge). SQLite transaction isolation guarantees exactly one worker succeeds. If the claim fails (race lost), return to Step 1 with the losing candidate excluded.
📋 Step 3: Build
Construct the prompt from the bead's context: title, body, workspace path, relevant files, and any dependency context. The prompt is a deterministic function of the bead state — same bead, same prompt.
🚀 Step 4: Dispatch
Load the agent adapter configuration (YAML), render the invoke template with the built prompt, and execute via bash -c. The agent runs headless — it receives a prompt, does work, and exits.
⏳ Step 5: Execute
The agent runs. NEEDLE waits. The only inputs are the exit code and stdout/stderr. There is no interactive communication during execution.
📊 Step 6: Outcome
Evaluate the result and follow the explicit path for the observed outcome:
| Outcome | Exit Code | Handler |
|---|---|---|
| ✅ Success | 0 | Validate output → close bead → log effort → loop |
| ❌ Failure | 1 | Log failure reason → release bead → increment retry count → loop |
| ⏰ Timeout | 124 | Release bead → mark deferred → loop |
| 💀 Crash | >128 | Release bead → create alert bead → loop |
| 🏁 Race Lost | 4 | (Handled at Step 2) → exclude candidate → retry select |
| 🫙 Queue Empty | — | Enter strand escalation → explore / mend / knot |
Every row is implemented. There are no unhandled cases.
🧶 Strand Escalation
When the primary workspace has no claimable beads, NEEDLE follows a strand sequence to find or create work. Each strand is evaluated in order — the first strand that yields a bead wins.
| # | Strand | Agent? | Purpose |
|---|---|---|---|
| 1 | 🪡 Pluck | Yes | Process beads from the assigned workspace |
| 2 | 🔧 Mend | No | Cleanup: orphaned claims, stale locks, health checks |
| 3 | 🔭 Explore | No | Search other workspaces for claimable beads |
| 4 | 🕸️ Weave | Yes | Create beads from documentation gaps (opt-in) |
| 5 | 🪢 Unravel | Yes | Propose alternatives for HUMAN-blocked beads (opt-in) |
| 6 | 💓 Pulse | Yes | Codebase health scans, auto-generate beads (opt-in) |
| 7 | 🪞 Reflect | Yes | Consolidate learnings from recent beads (opt-in) |
| 8 | 🪡 Splice | No | Document worker failures, create alert beads |
| 9 | 🪢 Knot | No | All strands exhausted — alert human, wait |
⚡ Parallel Workers
Multiple NEEDLE workers run independently with no central orchestrator. Coordination happens through the shared bead queue:
- Atomicity — the bead CLI's claim command uses SQLite transactions; exactly one worker wins each claim
- Determinism — all workers compute the same priority order; races are resolved by the database, not by timing
- Independence — each worker is a self-contained loop in its own tmux session
- Naming — workers use NATO alphabet identifiers:
alpha,bravo,charlie, ...
needle-claude-sonnet-alpha ──┐
needle-claude-sonnet-bravo ──┤
needle-codex-gpt4-charlie ───┼──► Shared .beads/ (SQLite + checkpoint)
needle-opencode-qwen-delta ──┤
needle-aider-sonnet-echo ────┘
🏗️ Supported Agents
NEEDLE is agent-agnostic. Any CLI that accepts a prompt and exits works.
| Agent | CLI | Input Method | Notes |
|---|---|---|---|
| Claude Code (interactive) | claude-interactive | stdin | Recommended — uses subscription billing; see plugin |
| Claude Code (API) | claude --print | stdin | Uses programmatic/API billing |
| OpenCode | opencode | file | |
| Codex CLI | codex | args | |
| Aider | aider --message | args | |
| Custom | any | configurable via YAML adapter |
Adding a new agent requires only a YAML configuration file — no code changes.
🔌 claude-interactive Plugin
The claude-interactive plugin ships as a separate release asset. It wraps the Claude Code CLI in a PTY so workers run under your Claude subscription instead of consuming programmatic API credits.
How it works: NEEDLE pipes subprocess stdio, which causes claude to detect a non-TTY and switch to API billing. claude-interactive creates an internal PTY so claude sees a real terminal, keeping it in interactive/subscription mode.
Install:
# Download the latest claude-interactive release
gh release download --repo jedarden/NEEDLE --pattern 'claude-interactive*'
chmod +x claude-interactive-install.sh
./claude-interactive-install.sh
Requirements: Python 3.10+, pyte (pip install pyte), and the claude CLI on PATH.
Run:
cd /path/to/workspace
needle run --agent claude-interactive --count 4
Source lives in plugins/claude-interactive/.
📁 Repository Structure
NEEDLE/
├── Cargo.toml # Rust crate manifest
├── install.sh # One-line installer for prebuilt binaries
├── plugins/
│ └── claude-interactive/ # PTY wrapper — subscription billing adapter for Claude Code
├── src/
│ ├── main.rs # Worker entry point
│ ├── lib.rs # Library root
│ ├── agent_event.rs # Agent event telemetry utilities
│ ├── claude_md_placement.rs # CLAUDE.md placement logic
│ ├── commit_hook.rs # Bead-Id trailer injection for git commits
│ ├── routing.rs # Model-based adapter routing
│ ├── bead_store/ # Abstract bead backend interface
│ ├── bin/ # Auxiliary binaries (transform helpers)
│ ├── canary/ # Release channel promotion, canary tests
│ ├── claim/ # Atomic bead claiming via SQLite transactions
│ ├── cli/ # Command-line interface parsing
│ ├── config/ # `.needle.yaml` parsing and defaults
│ ├── cost/ # Token + USD spend tracking per bead and worker
│ ├── decision/ # Decision point detection, ADR management
│ ├── dispatch/ # Agent invocation + YAML adapter loading
│ ├── drift/ # Session similarity, clustering, divergence detection
│ ├── health/ # Liveness, stale-claim cleanup, watchdog
│ ├── learning/ # Retrospective extraction, learnings management
│ ├── mitosis/ # Child-aware bead splitting
│ ├── outcome/ # Explicit handler per outcome type
│ ├── peer/ # Multi-worker coordination, peer discovery
│ ├── prompt/ # Deterministic prompt construction from bead
│ ├── rate_limit/ # Provider/model concurrency and RPM rate limiting
│ ├── registry/ # Worker state registry
│ ├── sanitize/ # Output redaction (gitleaks integration)
│ ├── skill/ # Skill library, retrieval, promotion
│ ├── span/ # W3C trace context utilities
│ ├── stats/ # Aggregation engine, A/B comparison
│ ├── strand/ # Pluck / Mend / Explore / Weave / Unravel / Pulse / Reflect / Splice / Knot
│ ├── supervisor/ # Fleet supervisor daemon (auto-scale)
│ ├── telemetry/ # OTLP exporter, gen_ai semantic conventions
│ ├── trace/ # Trace capture, storage, retention
│ ├── transcript/ # Session JSONL parsing, action-outcome extraction
│ ├── types/ # Shared types, error definitions
│ ├── upgrade/ # Self-update, hot-reload, rollback
│ ├── validation/ # Pre-dispatch and post-execution checks
│ └── worker/ # Worker session and identity management
├── tests/ # Integration tests
├── ci/ # Docker images used by CI (runs on Argo Workflows)
├── config/ # Vendored gitleaks rules
└── docs/ # Plan, research, examples, post-mortems
🔍 Observability
NEEDLE emits structured telemetry for every state transition, claim attempt, dispatch, and outcome. A silent worker is a broken worker.
Exported Signals
| Signal | Description |
|---|---|
| Traces | Spans for worker.session, bead.lifecycle, bead.claim, agent.dispatch, strand.evaluated, outcome.handled |
| Metrics | needle.beads.completed, needle.beads.duration, needle.agent.tokens.input, needle.cost.usd, and more |
| Logs | All events not represented as spans, with severity mapping (ERROR for failures, WARN for stale peers) |
OpenTelemetry (OTLP) Export
NEEDLE can export telemetry to any OpenTelemetry-compatible backend (Jaeger, Tempo, Grafana, Honeycomb, Datadog, etc.) via OTLP.
Minimal configuration (.needle.yaml):
telemetry:
otlp_sink:
enabled: true
endpoint: "http://localhost:4317" # gRPC, or :4318 for HTTP
protocol: "grpc"
Semantic conventions: NEEDLE follows OpenTelemetry's gen_ai.* semantic conventions for LLM telemetry, enabling out-of-the-box integration with GenAI dashboards (Grafana GenAI app, Langfuse, Honeycomb AI, etc.).
See docs/plan/plan.md for the complete semantic mapping table.
📊 Production Status
NEEDLE currently powers my own headless multi-agent workflow — workers run continuously against shared bead queues, dispatching to Claude Code and other CLIs, with full OTLP telemetry wired through. APIs are stable enough that I rebuild on top of them daily.
This is alpha software in the sense that I'm the primary user, not in the sense that "it doesn't work." Resource governance is delegated to claude-governor; session monitoring is handled by ccdash.
If you want to run NEEDLE in your own workflow, open an issue and I'll help.
🔗 Related Projects
- claude-governor — caps API spend and enforces weekly Anthropic quotas across NEEDLE worker fleets
- ccdash — TUI for monitoring Claude Code sessions, token usage, and worker activity
- CLASP — drop-in proxy letting Claude Code target OpenAI, Gemini, Anthropic, or any LLM backend
- agentists-quickstart — opinionated DevPod workspaces for running Claude Code + NEEDLE
📄 License
MIT
Part of jedarden.com · Read the write-up: jedarden.com/projects/needle/
This GitHub repo is a read-only mirror of git.ardenone.com/jedarden/NEEDLE — issues and PRs are welcome here either way.