Lemon Agent Guide

May 20, 2026 · View on GitHub

Effective agent context for the Lemon AI assistant platform.
Lemon is a local-first assistant and coding agent system with a multi-engine architecture supporting Claude, Codex, OpenCode, Pi, and native Lemon engines.

If you want to...	Look in...
Add/modify AI provider support	`apps/ai/`
Work on AI runtime auth facade boundary	`apps/lemon_ai_runtime/`
Work on coding tools or session management	`apps/coding_agent/`
Modify Telegram/Discord channel adapters	`apps/lemon_channels/`
Modify SMS/voice transports	`apps/lemon_gateway/`
Add new messaging channel adapters (X, XMTP, etc.)	`apps/lemon_channels/`
Work on agent routing or message flow	`apps/lemon_router/`
Build HTTP/WebSocket API features	`apps/lemon_control_plane/`
Manage configuration, secrets, or storage	`apps/lemon_core/`
Build reusable simulation harnesses	`apps/lemon_sim/`
Work with CLI runners/subagent spawning	`apps/agent_core/`
Create or modify skills	`apps/lemon_skills/`
Build cron jobs or automation	`apps/lemon_automation/`
Manage long-running external processes	`apps/lemon_services/`
Work on the web UI	`apps/lemon_web/`
Debug coding agent via RPC	`apps/coding_agent_ui/`
Browser automation via CDP/Playwright	`clients/lemon-browser-node/`

Parallel Work & Git Worktrees

When working on multiple tasks in parallel (either as the same agent or multiple agents), use git worktrees to avoid file editing conflicts. Store all worktrees under .worktrees/ in the repository root.

Workflow:

Create a worktree for each parallel task:

mkdir -p .worktrees
git worktree add .worktrees/task-name -b task-name
cd .worktrees/task-name

Work in isolation — Each worktree is an independent working directory backed by the same repo:
```
git status
```

Clean up when complete — After the branch is merged/closed, remove the worktree:

cd /path/to/main/lemon
git worktree remove .worktrees/task-name
git branch -d task-name

Why git Worktrees?

No file editing conflicts — Multiple agents can edit different files simultaneously without stepping on each other
Clean build contexts — Each worktree maintains separate _build/ and deps/ as needed
Easy cleanup — Remove worktrees when done without affecting the main checkout

Golden Rule:

Never have multiple agents editing the same working directory simultaneously. Always use separate worktrees for parallel tasks.

Agent Team Composition

When spawning agents for parallel work, match the agent tier to the task complexity. Don't use Opus for investigation or Sonnet for architectural decisions.

Role Model

Role	Internal Model	External Model	Typical Tasks
Junior/Mid Dev	Sonnet	Kimi	Investigation, plan file creation, test running, config cleanup, doc updates, dependency audits, simple refactors
Senior Dev	Opus	—	Complex refactoring, architectural extraction, correctness-critical code, multi-module decomposition
Staff Engineer	Codex (MCP)	—	Plan ownership/review, architectural oversight, cross-cutting design decisions, final validation

Guidelines

Default to the lowest tier that can do the job — Use Sonnet for exploration and investigation. Only escalate to Opus when the task involves complex logic, cross-module refactoring, or correctness-critical code.
Codex owns plans — Matches the existing owner: codex / reviewer: codex convention in the planning system. Codex reviews architectural decisions and validates decomposition strategies.
Escalation pattern: Sonnet investigates → Opus implements → Codex reviews. Not every task needs all three tiers.
Kimi for external/security: Security audits, pre-push hooks, and external review tasks (already established in the pre-push hook workflow).
Planning metadata: owner: and reviewer: fields in plan YAML front matter should reference these roles (e.g., owner: codex, reviewer: codex).

Spawning Examples

# Sonnet for investigation (junior/mid)
# Use model: "sonnet" in Task tool or --model sonnet in CLI

# Opus for complex implementation (senior)
# Use model: "opus" in Task tool or default CLI model

# Codex for plan review (staff)
# Use mcp__codex__codex tool with architectural review prompt

# Kimi for security audit (external)
# Use kimi CLI runner for pre-push or security review

Documentation Contract ⚠️

Work is not complete until it is adequately documented.

Any code change must be accompanied by updates to all relevant documentation. This is non-negotiable. Outdated documentation is technical debt that compounds and confuses future developers (including yourself).

When You Modify Code, You MUST:

Update AGENTS.md files — If you change architecture, patterns, dependencies, or behaviors described in any AGENTS.md, update it immediately.
Update README.md files — If your change affects setup, usage, APIs, or public interfaces, update the relevant README.
Update architecture docs in docs/ — If your change affects design decisions, addendums to existing docs, or new architectural patterns, update or add docs.
Update inline comments — Complex logic, public functions, and non-obvious behaviors must have accurate, up-to-date comments.
Update configuration examples — If you add/remove config options, update .lemon/config.toml examples and config documentation.

Examples of Documentation Debt to Avoid:

Changing a module's behavior without updating its @moduledoc or AGENTS.md
Adding a new tool/config/API without documenting how to use it
Refactoring architecture while leaving stale dependency diagrams
Modifying environment variables without updating .env.example or docs
Changing a behavior but leaving old instructions in guides

The Golden Rule:

If you changed how something works, you must change the documentation that describes how it works. No exceptions.

Future agents (and humans) depend on accurate documentation to be effective. Don't make their job harder by leaving stale docs.

Project Structure

apps/
├── agent_core/          # Core agent runtime, CLI runners (claude, codex, pi, kimi, opencode), subagent management
├── ai/                  # AI provider abstraction (Anthropic, OpenAI, Google, Azure, Bedrock)
├── lemon_ai_runtime/     # Thin auth/config facade boundary (OAuth resolver delegating modules)
├── coding_agent/        # Main coding agent with 35+ tools, session management, budget enforcement
├── coding_agent_ui/     # Thin wrapper that exposes coding_agent via RPC (mostly empty, used for tooling)
├── lemon_automation/    # Cron jobs, heartbeat manager, run submitter
├── lemon_channels/      # Channel adapters for inbound/outbound delivery (Telegram, Discord, X API, XMTP)
├── lemon_control_plane/ # HTTP/WebSocket API server with 112+ JSON-RPC methods
├── lemon_core/          # Shared primitives: config, store (ETS/JSONL/SQLite), secrets, PubSub bus
├── lemon_gateway/       # Gateway engines (claude, codex, pi, opencode, lemon, echo), voice/email/webhook/farcaster transports
├── lemon_mcp/           # MCP (Model Context Protocol) server/client bridge for CodingAgent tools
├── lemon_router/        # Message routing, agent directory, run orchestration
├── lemon_services/      # Long-running external process management (OTP-based, no umbrella deps)
├── lemon_sim/           # Reusable simulation harness primitives (projector/updater/action-space contracts)
├── lemon_skills/        # Skill registry, discovery, installation
└── lemon_web/           # Phoenix LiveView web interface

clients/
├── lemon-browser-node/  # Browser automation node via CDP/Playwright (TypeScript)
├── lemon-cli/           # Python TUI client packaged with uv
├── lemon-tui/           # Terminal UI client (TypeScript)
└── lemon-web/           # Web workspace (shared, server, web packages)

docs/                    # Architecture docs, design decisions
config/                  # Umbrella configuration (config.exs, runtime.exs, dev.exs, prod.exs)
scripts/                 # Utility scripts

Build, Test & Development

Elixir Umbrella

# Install dependencies
mix deps.get

# Compile all apps
mix compile

# Canonical repo-level test lanes
scripts/test help
scripts/test fast       # compile with warnings as errors + ExUnit excluding integration
scripts/test quality    # Lemon quality gates and focused quality tests
scripts/test clients    # Python CLI + Node client CI parity checks
scripts/test eval-fast  # small eval harness run
scripts/test smoke      # CI-only product-smoke pointer
scripts/test all        # useful local aggregate
scripts/test path apps/ai/test --seed 1

# Format code
mix format

TUI Client (TypeScript)

cd clients/lemon-tui
npm install
npm run build
npm run test:coverage
npm run dev      # Watch mode

Python CLI Client

cd clients/lemon-cli
uv sync --locked --dev
uv run ruff check src tests
uv run pytest
uv build --sdist --wheel

Web Client

cd clients/lemon-web
npm install
npm run dev      # Start web server + frontend
npm run build    # Build shared/server/web packages
npm run test:coverage

Browser Node Client (TypeScript)

cd clients/lemon-browser-node
npm install
npm run build
npm run test:coverage
npm run dev      # Watch mode

Quick Dev Bootstrap

./bin/lemon-dev    # Installs deps, builds, launches TUI
./bin/lemon        # Unified runtime (gateway + control plane + router + channels + web)
./bin/lemon send --to telegram:<chat_id> "done"  # Script notification to Telegram/Discord
./bin/lemon send --to discord:#ops --attach report.txt --attach trace.log "done"  # Upload script artifacts
./bin/lemon send --dry-run --to discord:#ops --attach report.txt "done"  # Validate without delivery
./bin/lemon-tui    # TUI attached to unified runtime; auto-starts runtime if needed

On Linux and other non-keychain environments, keep ~/.lemon/secrets_master_key as the canonical local master key file. ./bin/lemon now normalizes LEMON_SECRETS_MASTER_KEY from that file at startup so stale inherited shell env does not break provider or transport secret decryption.

Architecture Overview

Message Flow

[User via Telegram / Discord / XMTP / X / SMS]
    ↓
[lemon_channels or legacy gateway ingress] - Transport adapters and inbound normalization
    ↓
[lemon_router] - Route to appropriate agent, run orchestration, queue semantics
    ↓
[lemon_gateway] - Execution slots and engine lifecycle
    ↓
[coding_agent] - Execute tools, manage sessions, budget enforcement
    ↓
[agent_core] - CLI runners (claude/codex/pi/kimi/opencode), subagent spawning
    ↓
[ai] - LLM provider calls (Anthropic, OpenAI, Google, Azure, Bedrock)

Outbound message delivery goes through lemon_channels (Telegram, X API, XMTP adapters). The control plane (lemon_control_plane) provides the JSON-RPC API used by TUI/web clients.

Key Dependencies Between Apps

Derived from mix.exs files and enforced by mix lemon.quality (architecture boundary check):

lemon_control_plane ──→ lemon_core, lemon_router, lemon_channels, lemon_skills, lemon_automation, ai, lemon_ai_runtime, coding_agent*
lemon_router ─────────→ lemon_core, lemon_channels, coding_agent, agent_core
lemon_gateway ────────→ lemon_core, agent_core, coding_agent, lemon_channels*
lemon_automation ─────→ lemon_core, lemon_router, lemon_skills
lemon_channels ───────→ lemon_core, lemon_ai_runtime
coding_agent ─────────→ lemon_core, agent_core, ai, lemon_ai_runtime, lemon_skills
agent_core ───────────→ lemon_core, ai
lemon_ai_runtime ─────→ ai, lemon_core
lemon_mcp ────────────→ coding_agent, agent_core
lemon_sim ────────────→ lemon_core, agent_core, ai, lemon_ai_runtime
lemon_sim_ui ─────────→ ai, lemon_core, lemon_sim
lemon_skills ─────────→ lemon_core, agent_core, ai, lemon_channels
lemon_web ────────────→ lemon_core, lemon_router, lemon_ai_runtime
lemon_services ───────→ (no umbrella deps - standalone OTP service manager)
coding_agent_ui ──────→ coding_agent
ai ───────────────────→ lemon_core

* = runtime: false (compile-time only dependency)

Configuration

User config: ~/.lemon/config.toml
Project config: .lemon/config.toml (in repo root)
Secrets: Managed via mix lemon.secrets.* tasks (set, list, delete, status, init)
Config inspection: mix lemon.config - show resolved runtime config
Store migration: mix lemon.store.migrate_jsonl_to_sqlite

Key env vars:

ANTHROPIC_API_KEY - Claude provider
OPENAI_API_KEY - OpenAI provider
LEMON_LOG_LEVEL - Log level (debug/info/warning/error)
LEMON_STORE_PATH - Persistent store path
LEMON_WEB_ACCESS_TOKEN - Web UI auth token
LEMON_WEB_HOST / LEMON_WEB_PORT - Web server binding (prod)
LEMON_WEB_SECRET_KEY_BASE - Required in prod
LEMON_GATEWAY_HEALTH_PORT / LEMON_ROUTER_HEALTH_PORT - Health server port overrides for local parallel runtimes
LEMON_TELEGRAM_DEFAULT_CHAT_ID / LEMON_DISCORD_DEFAULT_CHANNEL_ID - Optional env overrides for ./bin/lemon send; config fallbacks live in [gateway.telegram] default_chat_id/default_thread_id/default_topic_id and [gateway.discord] default_channel_id/default_thread_id
DEEPGRAM_API_KEY - Speech-to-text
ELEVENLABS_API_KEY / ELEVENLABS_VOICE_ID - TTS
TWILIO_ACCOUNT_SID / TWILIO_AUTH_TOKEN / TWILIO_PHONE_NUMBER - SMS

Key Patterns

Adding a New Tool

Create module in apps/coding_agent/lib/coding_agent/tools/
Implement the tool pattern (see existing tools in the CodingAgent.Tools.* namespace)
Add to CodingAgent.Tools registry
Update tool policy if needed

Adding an AI Provider

Create provider module in apps/ai/lib/ai/providers/
Implement Ai.Provider behaviour
Register in Ai.ProviderRegistry

Adding a Gateway Transport

External channel adapters live in apps/lemon_channels/. Current adapters include Telegram and Discord. Gateway-native transports remain in apps/lemon_gateway/ (SMS/Twilio, voice, email/webhook/farcaster glue).

Create transport module in apps/lemon_gateway/lib/lemon_gateway/
Implement appropriate behaviour (see existing transports for patterns)
Wire up in LemonGateway.Application

Adding a Gateway Engine

Engines are in apps/lemon_gateway/lib/lemon_gateway/engines/. Current: claude.ex, codex.ex, pi.ex, lemon.ex, opencode.ex, echo.ex, cli_adapter.ex.

Create engine module implementing LemonGateway.Engine behaviour
Register in engine registry

Testing & Debugging

Gateway Debugging (Telegram)

# Terminal 1: Start gateway with debug logs
LOG_LEVEL=debug ./bin/lemon-gateway --debug --sname lemon_gateway_debug

# Terminal 2: Attach to BEAM node
iex --sname lemon_attach --cookie lemon_gateway_dev_cookie \
    --remsh lemon_gateway_debug@$(hostname -s)

Useful runtime checks:

# Scheduler state
:sys.get_state(LemonGateway.Scheduler)

# Engine lock waiters
:sys.get_state(LemonGateway.EngineLock)

# Thread workers
DynamicSupervisor.which_children(LemonGateway.ThreadWorkerSupervisor)

# Session history
LemonCore.Store.get_run_history(session_key, limit: 10)

API keys (OpenAI, Anthropic, AWS, etc.)
Passwords and authentication tokens
Private keys (SSH, SSL, JWT secrets)
Database connection strings with credentials
Environment files (.env) containing secrets
Hardcoded secrets in configuration files

Installation:

./bin/install-security-hook

Usage:

The hook runs automatically on git push
If sensitive data is detected, the push is blocked
To bypass in emergencies: git push --no-verify
To uninstall: rm .git/hooks/pre-push

Note: The hook is not installed by default. Each developer must opt-in by running the install script.

Documentation Index

docs/architecture_boundaries.md - Dependency boundaries and allowed cross-app references
docs/config.md - Runtime configuration reference
docs/skills.md - Skill system documentation
docs/quality_harness.md - Quality checks and cleanup (mix lemon.quality, mix lemon.cleanup)
docs/testing.md - Canonical repo-level test lanes and CI parity guidance
docs/assistant_bootstrap_contract.md - Bootstrap contract
docs/context.md - Context management
docs/remote-cli-task-execution-plan.md - Planning note for remote codex/claude task execution over generic runner backends
docs/plans/2026-03-19-ai-boundary-extraction-plan.md - Plan for moving auth/config/storage ownership out of apps/ai before extracting ai into its own repo
docs/subagent-parent-questions.md - Design for subagent-to-parent clarification requests via a narrow ask_parent path
docs/missions.md - Reverse-engineered Factory Missions behavior and Lemon implementation spec
docs/missions_phase1_plan.md - Concrete Phase 1 implementation plan for Lemon Missions
docs/telemetry.md - Telemetry and observability
docs/extensions.md - Extension system
docs/beam_agents.md - BEAM agent architecture
docs/benchmarks.md - Performance benchmarks
docs/model-selection-decoupling.md - Model selection design
docs/agent-loop/ - Agent loop design docs
docs/testing/ - Testing guides
docs/tools/ - Tool documentation

Coding Conventions

Elixir: snake_case files, CamelCase modules
TypeScript: Follow workspace ESLint config
Format: Run mix format before committing
Tests: *_test.exs for Elixir, *.test.ts for TypeScript
Commits: Short, imperative style (Fix gateway timeout, chore: update docs)
Documentation: See Documentation Contract above — code changes require doc updates

App-Specific Guides

Each app has its own AGENTS.md with detailed context:

App	Location
agent_core	`apps/agent_core/AGENTS.md`
ai	`apps/ai/AGENTS.md`
coding_agent	`apps/coding_agent/AGENTS.md`
lemon_core	`apps/lemon_core/AGENTS.md`
lemon_gateway	`apps/lemon_gateway/AGENTS.md`
lemon_channels	`apps/lemon_channels/AGENTS.md`
lemon_router	`apps/lemon_router/AGENTS.md`
lemon_control_plane	`apps/lemon_control_plane/AGENTS.md`
lemon_sim	`apps/lemon_sim/AGENTS.md`
lemon_skills	`apps/lemon_skills/AGENTS.md`
lemon_automation	`apps/lemon_automation/AGENTS.md`
lemon_services	`apps/lemon_services/AGENTS.md`
lemon_web	`apps/lemon_web/AGENTS.md`
lemon_mcp	`apps/lemon_mcp/README.md` (no AGENTS.md yet)
coding_agent_ui	`apps/coding_agent_ui/AGENTS.md`

Last updated: 2026-04-27 (updated router/gateway architecture flow and dependency map)