Architecture

May 29, 2026 · View on GitHub

Architecture

How It Works
Session Lifecycle
MCP Tools
Progressive Disclosure
Memory Hygiene
Topic Key Workflow
Project Structure
CLI Reference

How It Works

Agent saving a memory via mem_save
The agent proactively calls mem_save after significant work — structured, searchable, no noise.

Engram trusts the agent to decide what's worth remembering — not a firehose of raw tool calls.

The Agent Saves, Engram Stores

1. Agent completes significant work (bugfix, architecture decision, etc.)
2. Agent calls mem_save with a structured summary:
   - title: "Fixed N+1 query in user list"
   - type: "bugfix"
   - content: What/Why/Where/Learned format
3. Engram persists to SQLite with FTS5 indexing
4. Next session: agent searches memory, gets relevant context

Session Lifecycle

Session starts → Agent works → Agent saves memories proactively
                                    ↓
Session ends → Agent writes session summary (Goal/Discoveries/Accomplished/Next Steps/Files)
                                    ↓
Next session starts → Previous session context is injected automatically

MCP Tools

Tool	Purpose
`mem_save`	Save a structured observation (decision, bugfix, pattern, etc.); best-effort captures process-local current prompt context when available unless `capture_prompt=false`
`mem_update`	Update an existing observation by ID
`mem_delete`	Delete an observation (soft-delete by default, hard-delete optional)
`mem_suggest_topic_key`	Suggest a stable `topic_key` for evolving topics before saving
`mem_search`	Full-text search across all memories
`mem_session_summary`	Save end-of-session summary
`mem_context`	Get recent context from previous sessions
`mem_timeline`	Chronological context around a specific observation
`mem_get_observation`	Get full content of a specific memory
`mem_save_prompt`	Save a user prompt for future context
`mem_stats`	Memory system statistics
`mem_session_start`	Register a session start
`mem_session_end`	Mark a session as completed
`mem_capture_passive`	Extract learnings from text output
`mem_merge_projects`	Merge project name variants into canonical name (admin)
`mem_current_project`	Detect project from cwd — never errors, recommended first call
`mem_doctor`	Run read-only operational diagnostics for project detection and store health
`mem_judge`	Record a verdict for a pending memory conflict surfaced by `mem_save`
`mem_compare`	Persist a semantic relation verdict between two existing observations

Progressive Disclosure (3-Layer Pattern)

Token-efficient memory retrieval — don't dump everything, drill in:

1. mem_search "auth middleware"     → compact results with IDs (~100 tokens each)
2. mem_timeline observation_id=42  → what happened before/after in that session
3. mem_get_observation id=42       → full untruncated content

Memory Hygiene

mem_save now supports scope (project default, personal and global also accepted)
mem_save also supports topic_key; with a topic key, saves become upserts (same project+scope+topic updates the existing memory)
mem_save supports capture_prompt (true by default). When the same MCP process lifecycle has current prompt context for the same project and session, it best-effort records that prompt alongside the observation. The prompt context must be fed before the later mem_save (typically via mem_save_prompt); mem_save still succeeds if context is unavailable or prompt capture fails. Automated saves such as SDD artifacts should pass capture_prompt=false.
Exact dedupe prevents repeated inserts in a rolling window (hash + project + scope + type + title)
Duplicates update metadata (duplicate_count, last_seen_at, updated_at) instead of creating new rows
Topic upserts increment revision_count so evolving decisions stay in one memory
mem_delete uses soft-delete by default (deleted_at), with optional hard delete
mem_search, mem_context, recent lists, and timeline ignore soft-deleted observations

Topic Key Workflow (Recommended)

topic_key turns mem_save into an upsert: if a memory with the same project + scope + topic_key already exists, the existing observation is updated in place (revision_count++) instead of creating a new row. Without a topic_key, every mem_save creates a new observation even when the content describes the same evolving topic.

Use topic keys for knowledge that changes over time: architecture decisions, long-running feature notes, recurring patterns, configuration choices. Skip them for one-off bugs, single facts, or anything that does not evolve.

Format convention

Topic keys follow slash-separated lowercase kebab-case:

family/specific-description

Examples:

architecture/auth-model
bug/nil-panic-in-user-list
decision/database-choice
pattern/error-handling-convention
config/ci-environment

Why this format? SQLite FTS5 tokenises on word boundaries. Lowercase kebab-case ensures the key fragments are individually searchable and do not create unexpected FTS5 token splits.

Anti-patterns to avoid:

Anti-pattern	Problem	Correct form
`authModel`	camelCase breaks FTS5 tokenisation	`architecture/auth-model`
`auth model`	spaces create accidental multi-token keys	`architecture/auth-model`
`ARCHITECTURE/AUTH`	uppercase is inconsistent with FTS5 normalisation	`architecture/auth-model`
`auth/model/v2/final`	more than 2 levels — use `v2` in the description	`architecture/auth-model-v2`
`bugfix`	no slash — looks like a family with no description	`bug/auth-nil-panic`

Decision table — when to use topic_key

Situation	Use topic_key?	Reasoning
Architecture or design decision that may evolve	Yes	Keeps history in one observation, incrementing `revision_count`
Long-running feature work (spans multiple sessions)	Yes	Single source of truth across sessions
A pattern or convention established for the project	Yes	One canonical entry, updated as the pattern matures
Bug fix that was self-contained and is now closed	No	A single observation is fine; no future updates expected
One-off discovery or fact	No	Creating a key you will never reuse adds noise
Multiple independent decisions on the same broad topic	No — use distinct keys	Different decisions must have different keys or they will overwrite each other

The mem_suggest_topic_key-first workflow

When you are not sure which key to use, call mem_suggest_topic_key before mem_save. It applies a family heuristic based on the observation type and title, returning a suggested key you can use directly or adjust:

1. mem_suggest_topic_key(type="architecture", title="Auth model")
   → returns: "architecture/auth-model"

2. mem_save(..., topic_key="architecture/auth-model")
   → creates new observation (revision_count=1)

3. (later session) mem_save(..., topic_key="architecture/auth-model")
   → updates existing observation (revision_count=2)

mem_suggest_topic_key families:

architecture/* — architecture, design, ADR-like observations
bug/* — bug fixes, regressions, panics, error root causes
decision/* — explicit decisions with tradeoffs
pattern/* — naming conventions, structural patterns, coding standards
config/* — configuration and environment setup
discovery/* — non-obvious findings about the codebase
learning/* — team knowledge and onboarding notes

If none of these families fit, it is usually fine to skip the key and let mem_save create a plain observation.

Hierarchical keys — max 2 levels

Keys are organisational only; there is no parent–child relationship in the store. Two levels (family/description) cover almost every case. Use the description segment to add specificity rather than adding more slashes:

architecture/auth-model          ✓ two levels, specific
architecture/auth-model-v2       ✓ version in description
architecture/auth/model/detail   ✗ three levels — flatten to two

Lifecycle and pruning

Topic keys are not pruned automatically. An observation updated via upsert keeps a single row with the latest content and an incremented revision_count. Use mem_delete to remove an observation (soft-delete by default) when a topic is no longer relevant. Soft-deleted observations are excluded from search and context but their IDs remain in the store for audit purposes. Use --hard to remove them permanently.

Scope interaction

topic_key upsert is scoped to project + scope + topic_key. The same key used with different scopes creates independent observations:

project=engram, scope=project, topic_key=architecture/auth-model  → observation A
project=engram, scope=personal, topic_key=architecture/auth-model → observation B (independent)

This means a personal note on the same topic does not overwrite the shared project observation.

Project Structure

engram/
├── cmd/engram/main.go              # CLI entrypoint
├── internal/
│   ├── store/store.go              # Core: SQLite + FTS5 + all data ops
│   ├── server/server.go            # HTTP REST API (port 7437)
│   ├── mcp/mcp.go                  # MCP stdio server (19 tools)
│   ├── setup/setup.go              # Agent plugin installer (go:embed)
│   ├── cloud/                       # Optional cloud runtime (Postgres + dashboard)
│   │   ├── cloudserver/             # /sync API + dashboard mount + auth/session bridge
│   │   ├── cloudstore/              # Cloud chunk storage and dashboard read-model queries
│   │   ├── dashboard/               # Server-rendered dashboard routes + embedded static assets
│   │   └── auth/                    # Bearer token auth + signed dashboard sessions
│   ├── project/                     # Project name detection + similarity matching
│   │   └── detect.go               # DetectProject, DetectProjectFull, 5-case algorithm
│   ├── sync/sync.go                # Git sync: manifest + compressed chunks
│   └── tui/                        # Bubbletea terminal UI
│       ├── model.go                # Screen constants, Model, Init()
│       ├── styles.go               # Lipgloss styles (Catppuccin Mocha)
│       ├── update.go               # Input handling, per-screen handlers
│       └── view.go                 # Rendering, per-screen views
├── plugin/
│   ├── opencode/engram.ts          # OpenCode adapter plugin
│   └── claude-code/                # Claude Code plugin (hooks + skill)
│       ├── .claude-plugin/plugin.json
│       ├── .mcp.json
│       ├── hooks/hooks.json
│       ├── scripts/                # session-start, post-compaction, subagent-stop, session-stop
│       └── skills/memory/SKILL.md
├── skills/                         # Contributor AI skills (repo-wide standards + Engram-specific guardrails)
├── setup.sh                        # Links repo skills into .claude/.codex/.gemini (project-local)
├── assets/                         # Screenshots and media
├── DOCS.md                         # Full technical documentation
├── CONTRIBUTING.md                 # Contribution workflow and standards
├── go.mod
└── go.sum

CLI Reference

engram setup [agent]      Install/setup agent integration (opencode, claude-code, gemini-cli, codex)
engram serve [port]       Start HTTP API server (default: 7437)
engram mcp                Start MCP server (stdio transport)
engram tui                Launch interactive terminal UI
engram search <query>     Search memories
engram save <title> <msg> Save a memory
engram delete <obs_id>    Delete an observation [--hard] (soft-delete by default; --hard removes permanently)
engram delete session <id>
                          Delete a session by ID (session must have no observations)
engram delete prompt <id>
                          Delete a prompt by ID (permanent)
engram delete project <name> [--hard]
                          Cascade-delete a project: soft-deletes observations (or hard-deletes
                          with --hard, which also removes sessions); always removes prompts
engram timeline <obs_id>  Chronological context around an observation
engram context [project]  Recent context from previous sessions
engram stats              Memory statistics
engram export [file]      Export all memories to JSON
engram import <file>      Import memories from JSON
engram sync               Export new memories as compressed chunk to .engram/
engram sync --all         Export ALL projects (ignore directory-based filter)
engram sync --cloud --project <name>
                          Sync against configured cloud endpoint (project-scoped)
engram conflicts <sub>    Inspect and manage memory conflict relations
                            list, show, stats, scan, deferred
engram doctor             Run read-only operational diagnostics [--json] [--project P] [--check CODE]
engram cloud status       Show cloud runtime/config status
engram cloud config --server <url>
                          Configure cloud server URL
engram cloud enroll <project>
                          Enroll a project for cloud sync
engram cloud serve        Run cloud backend + dashboard
engram cloud upgrade <doctor|repair|bootstrap|status|rollback> --project <name>
                          Guided upgrade workflow for existing projects
engram projects list      Show all projects with obs/session/prompt counts
engram projects consolidate  Interactive merge of similar project names [--all] [--dry-run]
engram projects prune     Remove projects with 0 observations [--dry-run]
engram obsidian-export    Export memories to Obsidian vault (beta)
engram version            Show version

Local server auth:

ENGRAM_HTTP_TOKEN: optional Bearer auth for engram serve. When set, the following routes require Authorization: Bearer <token>: DELETE /sessions/{id}, DELETE /observations/{id}, DELETE /prompts/{id}, GET /export, POST /import, POST /projects/migrate. Comparison is constant-time; token is read per-request. When unset, all routes are open (zero-config default).
ENGRAM_TIMEZONE: IANA zone name for timestamp display in TUI and cloud dashboard (e.g. America/New_York). Falls back to system local when unset or invalid.

Cloud constraints (current behavior):

Cloud is opt-in replication/shared access; local SQLite remains source of truth.
engram cloud serve requires ENGRAM_CLOUD_ALLOWED_PROJECTS in both token-auth and insecure no-auth mode. Use * to allow all projects (dev/internal deploys) — bypasses per-project name enforcement while still requiring a non-empty project on each request.
Authenticated cloud serve requires ENGRAM_CLOUD_TOKEN + explicit non-default ENGRAM_JWT_SECRET.
Insecure local-dev mode (ENGRAM_CLOUD_INSECURE_NO_AUTH=1) still requires the project allowlist and must not be used in production.

Cloud route/auth split (current behavior):

Local runtime (engram serve) exposes local JSON APIs and GET /sync/status only.
Cloud runtime (engram cloud serve) exposes GET /health, GET /sync/pull, GET /sync/pull/{chunkID}, POST /sync/push, and /dashboard/*.
Dashboard public routes: GET /dashboard/health, GET/POST /dashboard/login, POST /dashboard/logout, GET /dashboard/static/*.
Dashboard protected routes: GET /dashboard, /dashboard/stats, /dashboard/activity, /dashboard/browser (/observations, /sessions, /sessions/{sessionID}, /prompts), /dashboard/projects, /dashboard/projects/list, /dashboard/projects/{project}, /dashboard/projects/{name}/observations|sessions|prompts, /dashboard/contributors, /dashboard/contributors/list, /dashboard/contributors/{contributor}, /dashboard/admin, /dashboard/admin/projects, /dashboard/admin/users, /dashboard/admin/users/list, /dashboard/admin/health, POST /dashboard/admin/projects/{name}/sync, /dashboard/sessions/{project}/{sessionID}, /dashboard/observations/{project}/{sessionID}/{syncID}, /dashboard/prompts/{project}/{sessionID}/{syncID}.
Note: /dashboard/admin/contributors was removed; user/contributor management lives under /dashboard/admin/users.
In authenticated mode, protected dashboard routes require a signed dashboard cookie (obtained via /dashboard/login + bearer token) and do not accept direct bearer headers as a browser session substitute.
In insecure mode (ENGRAM_CLOUD_INSECURE_NO_AUTH=1 with no bearer token), dashboard auth is bypassed and /dashboard/login redirects to /dashboard/.

Dashboard visual-parity layer

The cloud dashboard (internal/cloud/dashboard/) is rendered server-side using templ components. This section documents the key invariants introduced in the cloud-dashboard-visual-parity change.

Principal bridge

Display name is surfaced through MountConfig.GetDisplayName func(r *http.Request) string. If nil or if the closure returns an empty string, all handlers fall back to "OPERATOR". The bridge is implemented in internal/cloud/dashboard/principal.go and accessed via h.principalFromRequest(r). Handlers never read r.Context() for identity — they read the MountConfig closures.

Push-path pause guard

A per-project sync pause is stored in cloud_project_controls (Postgres). The POST /sync/push handler in cloudserver.go checks IsProjectSyncEnabled(project) immediately after authorizeProjectScope succeeds, using a structural interface assertion. A paused project returns HTTP 409 Conflict with error_code: "sync-paused". This is enforced server-side — the admin toggle is never purely cosmetic. Regression guard: TestPushPathPauseEnforcement in cloudserver_test.go.

Audit log boundary (structural interface pattern)

When a push is rejected due to a paused project, both handleMutationPush and handlePushChunk emit an audit entry via cloudstore.InsertAuditEntry. The audit capability is accessed through a structural type assertion against an anonymous interface:

if auditor, ok := s.store.(interface {
    InsertAuditEntry(ctx context.Context, entry cloudstore.AuditEntry) error
}); ok {
    _ = auditor.InsertAuditEntry(r.Context(), cloudstore.AuditEntry{...})
}

Why ChunkStore and MutationStore are NOT extended: these interfaces are consumed by a wide surface of test fakes, integration adapters, and future clients. Adding InsertAuditEntry to them would require every implementer (real and fake) to implement a method that is only relevant to the rejection path. The structural assertion isolates the audit concern at the call site and makes it trivially optional: stores that do not implement InsertAuditEntry (e.g., legacy test fakes) silently skip the audit with a log warning — no panic, no 5xx.

Synchronous insert rationale: the insert is synchronous (no goroutine, no channel) because the 409 is already a fast path (the data was never stored). The ~1ms overhead of a Postgres INSERT is acceptable; the alternative (buffered async) would complicate recovery, lose entries on process restart, and add concurrency concerns with no meaningful latency benefit for the caller.

Audit boundary: audit entries are only emitted at the pause-rejection site. Pull requests (GET /sync/mutations/pull) never emit audit entries; paused projects continue to serve reads unrestricted.

Composite-ID URL scheme

Detail pages use composite path parameters because the integrated store is chunk-centric (no globally unique numeric IDs):

Page	URL pattern
Session detail	`GET /dashboard/sessions/{project}/{sessionID}`
Observation detail	`GET /dashboard/observations/{project}/{sessionID}/{syncID}`
Prompt detail	`GET /dashboard/prompts/{project}/{sessionID}/{syncID}`

Path values are extracted via r.PathValue(name) (Go 1.22 net/http.ServeMux). syncID values are validated as non-empty with len <= 128.

Insecure-mode regression guard

TestInsecureModeLoginRedirects in cloudserver_test.go asserts that GET /dashboard/login with auth == nil returns 303 to /dashboard/. This prevents silent regressions if the no-auth short-circuit path is modified.

Cloud Autosync Manager

internal/cloud/autosync/Manager is a lease-guarded background goroutine started by engram serve and engram mcp when ENGRAM_CLOUD_AUTOSYNC=1. It implements the local-first invariant: all network I/O happens in its own goroutine and never holds locks shared with the SQLite write path.

Data flow

Local Write → store.WriteObservation → [SQLite sync_mutations journal]
                                               ↓ (onWrite hook)
                                       Manager.NotifyDirty() [buffered-1 chan]
                                               ↓ (debounce 500ms)
                                       Manager.cycle()
                                         ├─ AcquireSyncLease (SQLite)
                                         ├─ push: ListPendingSyncMutations → MutationTransport.PushMutations
                                         │         → POST /sync/mutations/push → cloudstore.InsertMutationBatch
                                         └─ pull: GetSyncState → MutationTransport.PullMutations
                                                   → GET /sync/mutations/pull (filtered by enrollment)
                                                   → store.ApplyPulledMutation
                                               ↓
                                       autosyncStatusAdapter → /sync/status → dashboard pill

Lease semantics

The Manager holds a SQLite-backed lease during each cycle. StopForUpgrade sets PhaseDisabled and does NOT release the lease, so no other worker can pick up sync during an upgrade window. ResumeAfterUpgrade clears the disabled flag and re-arms the loop without restarting the process.

Mutation endpoints

The cloud server exposes two routes registered in cloudserver.go and handled by mutations.go:

Method	Path	Purpose
`POST`	`/sync/mutations/push`	Accept a batch of up to 100 mutations from the client
`GET`	`/sync/mutations/pull`	Return mutations since a cursor, filtered by caller's enrolled projects

Both require Authorization: Bearer <token>. Push enforces the project-level sync pause (HTTP 409 on sync_enabled=false) and the configured cloud push body limit (ENGRAM_CLOUD_MAX_PUSH_BYTES, default 8 MiB). Pull filters server-side.

Next Steps

Agent Setup — connect your agent to Engram
Plugins — what the OpenCode and Claude Code plugins add
Obsidian Brain — visualize memories as a knowledge graph (beta)