Architecture

September 18, 2026 · View on GitHub

This document is for people changing deja internals.

Source parsers

Parsers live in internal/sources and return []model.Session. The table is the thirty-three the loader registers; docs/registry/ describes each store's layout in detail, and internal/sources/registry_test.go checks that index against the loader list.

SourceCodeInput
Claude Codeclaude.goJSONL files under ~/.claude/projects
Codex CLIcodex.gorollout JSONL files plus history.jsonl under ~/.codex
opencodeopencode.goSQLite database at ~/.local/share/opencode/opencode.db
aideraider.go.aider.chat.history.md files under configured project roots
Ampamp.goone JSON thread object per ~/.local/share/amp/threads/*.json
Gemini CLIgemini.goJSON and JSONL chats under ~/.gemini/tmp
Cursorcursor.goSQLite state stores plus CLI agent transcripts
Antigravityantigravity.goJSONL transcripts under ~/.gemini/antigravity*
Grok Buildgrok.goACP update streams and session summaries under ~/.grok/sessions
Qwen Codeqwen.goJSONL chats under ~/.qwen/projects/*/chats
pipi.goJSONL transcripts under ~/.pi/agent/sessions
Copilot CLIcopilot.goevents.jsonl per session under ~/.copilot/session-state
VS Code Copilot Chatcopilot_chat.go.jsonl / .json under VS Code User workspaceStorage/*/chatSessions
Clinecline.gotask JSON under the VS Code extension's storage, both store generations
Continuecontinuedev.goone JSON document per session under ~/.continue/sessions, with sessions.json as the list
Roo Coderoo.gotask JSON under rooveterinaryinc.roo-cline in VS Code globalStorage, and the CLI's own store under ~/.vscode-mock/global-storage
Goosegoose.golegacy JSONL sessions and the newer SQLite session store
Kimi Codekimi.goper-agent wire.jsonl under ~/.kimi-code/sessions
OpenClawopenclaw.goappend-only pi-format JSONL under ~/.openclaw/agents
omp (Oh My Pi)omp.goJSONL transcripts under ~/.omp/agent/sessions and each profile beside it
prime-agent (PrimeIntellect)prime.goJSONL transcripts under ~/.prime/agent/sessions
DeepSeek Harnessdeepseek.gozstd-compressed session JSONL under ~/.dsh/sessions
Zedzed.gothreads in the SQLite store at Zed/threads/threads.db
Crushcrush.goSQLite databases named by projects.json, plus <project>/.crush/crush.db
Cherry Studiocherrystudio.goClaude-format JSONL under the app's Data/Agents/.claude/projects
Kilo Codekilo.gotask JSON under the VS Code extension's storage, plus the CLI's kilo.db
Kirokiro.goCLI JSONL under ~/.kiro/sessions/cli, and messages.jsonl per IDE session
Command Codecommandcode.goJSONL under ~/.commandcode/projects
ZCodezcode.goJSONL under ~/.zcode/projects, plus the CLI's db.sqlite
gajae-codegjc.goJSONL under ~/.gjc/agent/sessions
Senpisenpi.goJSONL under ~/.senpi/agent/sessions
Kimchi Codingsenpi.goJSONL under the harness directory's sessions, one directory per encoded cwd
Hermeshermes.go, hermes_pg.goSQLite state per profile, or Postgres when DEJA_HERMES_PG_DSN is set
deja notesnotes.godeja remember entries in notes.jsonl

File-based sources are parsed with a worker pool sized to runtime.NumCPU(). Results are collected by input file index and then appended in sorted path order, so parsing can be parallel while index writes stay deterministic.

opencode and Cursor IDE state are read through the local sqlite3 command. Cursor CLI transcripts are plain JSONL. There is no CGO SQLite dependency.

Every one of those reads carries a wall-clock budget, ten minutes by default. One sqlite3 child once ran 13m54s with 0.75s of CPU in deja itself, and nothing in the tree set a deadline, so the run looked hung rather than slow. A store that runs out is an ordinary read error: the harness reports as unreadable, deja doctor names it, and the rest of the index still builds. DEJA_STORE_TIMEOUT takes a duration, and a zero or negative one turns the cap off for someone who would rather wait than lose a store.

Index format

Default path: ~/.cache/deja/index.db.

Files:

  • records.bin: length-prefixed records. Each record stores session key, source path, role, text, and timestamp. The session key, path and role are interned ids in the record's prefix, outside the body, so a scan for one kind of record reads the prefix and skips the rest; bodies of 8 KiB and up are deflated, smaller ones are not, because the read side pays more for a lower floor than the index saves.
  • buckets/*.bin: token bucket files. A token maps to compact postings: record offset, session ordinal, and one bit marking the posting as a work record.
  • manifest.gob / sessions.gob: index version, source file state, redaction counters, sync export watermarks, imported-record dedupe keys, session metadata (including ordinals, the files a session touched most, and hashes of the questions it asked), build time, and search scope.

Roles

Beyond user, assistant and developer, records carry what the agent did:

roleholds
tool-outputwhat a tool printed. Claude files this under the user role in its own transcripts, which is why it used to arrive labelled as something a person said
filesthe paths a turn opened or edited
commanda shell command worth keeping — an allowlist of build, test, VCS and deployment tooling, single-line only
edita span an edit replaced: the path on the first line, the exact removed bytes after it. Only the path earns postings, since nothing searches the body
summarya harness's own digest of a conversation it compacted away. Kept, because it is the only record of the turns that went, and not filed as speech: on one store 1,906 of them were 19.6% of everything indexed in the sessions that had them, and none of them ever won a quoted line for a real question

These are indexed and searchable by --role, and served in ordinary results only when asked for by role: a path that happens to contain the words of a question is not an answer to it. The postings carry a bit for them so the per-session read bound can spend its budget on speech first.

Notes source

Explicit notes are stored as one JSON object per line in ~/.local/share/deja/notes.jsonl, or under XDG_DATA_HOME; DEJA_NOTES_FILE overrides the path. Each record contains an RFC3339 ts, project, and text. Notes are grouped into one user-message session per project and calendar day in the local zone of whichever run indexed them, then redacted and indexed like every other source. deja index regroups buckets minted in another zone. The file is primary data; the index remains a rebuildable cache.

Secret redaction

internal/redact runs before every writeRecord path: cold rebuild, writeSessions, non-append incremental replacement, and append-only incremental ingest. The pass is disabled only when DEJA_NO_REDACT=1 is set; that escape hatch is unsafe because plaintext credentials will be written to the local index.

The redactor replaces only secret values, keeping keys and surrounding prose searchable. It covers AWS access keys and AWS secret assignments, generic credential assignments in ASCII and in other scripts, bearer tokens and JWTs, PEM and PGP private key blocks, provider token prefixes, connection URLs with scheme://user:pass@host credentials, credentials handed to a program on its command line (sshpass -p, mysql -pSecret, curl -u user:pass, --password and its siblings), netrc and cookie lines, bare high-entropy values in secret-shaped positions, and a password stated in prose — "the admin password is …", where no delimiter exists for the other rules to find.

Each rule that could match ordinary text carries a gate, and the gates are the part worth reading before adding a rule: the prose form requires the value to hold a digit or a symbol and to end at the first space, so "the password is wrong" and "the password is the same as staging" are left alone; the entropy rule excludes hex digests, UUIDs, paths and identifiers; the long command-line flags leave an ordinary word alone when the separator is a space rather than an =. docs/SECURITY-MODEL.md states the boundary these add up to.

Redaction counts are accumulated per source file in FileState.Redactions and as a manifest total. deja sources reads those counters and prints redacted= per harness.

Search flow:

  1. Tokenize the query.
  2. Read posting lists from the token buckets.
  3. Intersect posting lists for multi-word searches.
  4. Filter postings by session metadata and read every matching candidate record.
  5. Group records back into sessions and score them in internal/search with BM25 (k1=1.2, b=0.75). Document frequency and document length are measured over the candidate records at search time. User-message term contributions receive a 1.3 multiplier, and the score is multiplied by 1/(1+age_days).
  6. Sort by score, then updated time descending, then session ID ascending; the normal result limit is applied after ranking.

--harness, --project, and --since are applied from session metadata before scoring. --role is applied while reading candidate records.

Regex search scans records because arbitrary regex cannot use token postings safely.

deja blame retrieves candidates using the basename stem through the existing token postings, then verifies the basename as a path component or standalone word in the candidate text. Full and longer suffix path mentions outrank bare basenames; mention counts are blended with recency and an absolute project-root match receives a boost.

deja blame <path>:<line> answers about one line before that listing. Git names the commit that last changed the line; the session is the one whose edit record replaced the same text that commit shows as deleted, which is what makes the claim "this session performed this change" rather than "this session was open nearby". Spans are compared on whitespace-collapsed text and only from 24 runes up, since } and return nil are in every diff. Where no session matches, or git cannot answer, the line gets one sentence saying which silence it is — including the line count when the line is past the end of the file. No reason is printed with it: over 81 attributed lines the session's own conclusion overlapped the change 0 times, so a line lifted out of a session would read as the reason for a change it has nothing to do with.

Sync format

deja sync export <dir> reads records.bin and writes JSONL batch files named deja-sync-<source-hash>-<timestamp>.jsonl. Each line is one object:

{"harness":"claude","session_id":"abc123","project":"api","role":"assistant","text":"fixed by ...","time":"2026-07-14T12:00:00Z"}

The export watermark is per peer and source path (falling back to session key for synthetic records) and is stored in manifest.gob as the max exported record timestamp. Re-running export emits only records with a newer timestamp for that source and that peer, so what one machine has already received says nothing about what another still needs. An export with no peer named keeps the bare source key, which is what manifests written before per-peer watermarks carry. Text is redacted again during export.

deja sync import <dir> reads all *.jsonl batches, appends records to the local index, updates touched token buckets, and writes imported session metadata with the original harness and an imported: project prefix. Imported IDs are namespaced (imported-<hash>) so they do not clobber local sessions from the same harness. The manifest stores dedupe keys of harness:session_id:time, making re-import idempotent. Imported records live only in the index, so full rebuilds replay them from the old records.bin before regenerating from sources, and exports skip them to avoid echoing history back to its origin.

Which machines are one machine

The machines this one syncs with live in peers.json beside policy.json, outside the index, since who you sync with outlives any one rebuild. A bare deja sync walks that file and exchanges with every row, both ways.

A host is matched the way ssh matches it: case-insensitively, but only after the last @. box and BOX are one machine; Root@box and root@box are two logins on it. That rule decides three things at once — which row an exchange is recorded against, which rows a deja sync forget removes, and which watermark an export advances, so one machine cannot be sent its whole history twice under two spellings.

A run uses the spelling already stored for a host rather than the one typed, for the same reason.

The name a machine calls itself is not the alias you type at it, and an imported session is stamped with the former. A pull learns the pairing and records it on the peer's row, and deja doctor then counts what arrived from a machine under either name. A machine only ever pushed to has nothing to learn from, so it counts nothing — which is honest: nothing has arrived from it.

deja sync ssh <host> wraps the same export/import in one command: export to a temp dir, scp the batches, run the remote import (system ssh/scp, remote binary from PATH or ~/.local/bin/deja).

Incremental algorithm

currentFiles records path, size, and mtime for known stores.

EnsureForSearch compares the current file set with manifest.gob:

  • fresh manifest: do nothing;
  • version or scope mismatch: rebuild. Two versions are tracked: the content version, which moves when deja derives something new from a transcript, and the on-disk format, which moves only when an older layout would be mis-read. A store whose content version is stale is re-read but keeps answering from what it has; only a format mismatch makes search say it cannot answer yet;
  • append-only JSONL/opencode changes: append new records and update touched buckets;
  • removed files or non-append changes: rewrite the index while preserving unchanged records and replacing changed sessions.

A file takes the append path only when the prefix deja already read is still byte-for-byte what it read: a rewind that truncates and regrows past the old length looks exactly like growth, and appending onto it would leave the replaced turns in the index for good. The recorded prefix hash is what tells the two apart, so a live transcript that only grew is read from its last safe offset rather than reparsed whole.

Search does not wait for the rewrite branch. EnsureForSearchStale runs the cheap half inline and reports that the rest is outstanding; the caller refreshes in a detached warmup and answers from the index it already has. Both the MCP tools and the CLI take that path — a rewrite of a gigabyte of records and buckets is not a cost a query can pay — while deja index and deja search --rebuild still do the work in front of you.

Cold rebuild does all parsing first, then writes records.bin, buckets, and manifest from one goroutine. That keeps the on-disk index coherent and avoids concurrent writers. The four sidecars derived afterwards — the co-occurrence map, fix pairs, the recurring-command table and the failures — each write their own file and read nothing the others write, so they run together, and fix mining and the co-occurrence map also run per session across cores. That is most of the difference between a 51s and a 30s rebuild on a real store.

MCP server design

cmd/deja/mcp.go implements a small JSON-RPC stdio server.

Supported methods:

  • initialize
  • tools/list
  • tools/call

Tools:

  • recall: compact snippets for matching sessions.
  • recall_context: the markdown digest deja ctx prints.
  • blame: the sessions that discussed a file, and what was decided.
  • fix: what this machine ran after the same error last time.
  • how: the real invocation for a tool here, from what agents ran.
  • remember: stores one durable decision for later recall.

Each carries an annotation naming what it is for, so an agent can choose between them without reading the descriptions in full.

The MCP server calls the same index/search code as the CLI. It writes protocol responses to stdout and keeps logs/progress off stdout so agents receive valid JSON-RPC.

Claude SessionStart hook

deja install --auto installs the Claude MCP entry and adds a matcher-less command hook to ~/.claude/settings.json:

{"type":"command","command":"~/.config/deja/bin/deja-hook hook-context"}

The path in the entry is a launcher deja writes, not the binary it installed from. The launcher is three lines of shell that resolve the binary when the hook runs — DEJA_BIN, the path the install ran from, the PATH, then the usual install locations — so a release, a Homebrew upgrade or an npm install somewhere else moves the binary without any config being rewritten. On Windows there is no launcher and the entry holds the binary's path, because a .cmd cannot be exec'd the way a shebang script can.

deja hook-context is intentionally hidden from normal help. It first checks for a pending compaction recovery packet, which can exist before the first index build. For ordinary project recall, it derives the current project from the payload's cwd, else CLAUDE_PROJECT_DIR if the host exports one, else the directory it was run in, using the same Claude project-name logic as the parser, reads only an existing warm index (manifest.gob/sessions.gob must already exist), selects the most recent matching sessions by metadata project (ranked by the files the working tree is touching), leads them with the project's accepted promoted notes, and prints Claude's SessionStart response JSON with a compact markdown digest capped at 2KB. It never triggers a cold index build; missing index, empty results, corrupt data, or any other error produce no output and exit 0 so agent startup is not blocked. --plain prints the digest without the hook envelope, and --once gives it to the first turn of a session and nothing after — for a harness whose session-start output goes nowhere, where the digest has to ride the per-prompt hook instead (Kimi Code).

--auto wires four more hooks with the same best-effort contract. The ordinary recall paths use a warm index; compaction capture reads the current transcript and stores a bounded packet without building the search index:

  • hook-prompt (UserPromptSubmit) searches the index for the prompt's content and injects a small digest, or the you have been here line on a déjà-vu match. What it searches on is the ask — the last line ending in ?, or the last line carrying a word — read before the rest of the prompt, so a pasted repo listing or stack trace above the question does not spend the six-term budget. The same question from the same reader is answered once an hour: the per-session cooldowns count sessions shown, so an identical prompt on a timer used to walk one session further down the ranking on every tick. A spawned agent is exempt, because a fleet is many readers behind one id.
  • hook-tool (PreToolUse, matched to the editing/command tools) reads the tool payload — a Bash command or an Edit/Write/apply_patch target — and injects one line naming that file's or command's prior decision. Deliberately thin: it fires once per action, so it dedupes per agent session and carries at most one decision. A fact it has already given this session yields to the next one it can offer rather than ending the call — for a command that is the missing program, then the failure it hit here — but only among those two, because the producers behind them rank and load sessions and cost a hundred milliseconds at the point of action. How much history it waits for depends on what it has to say — a bare count of sessions needs five before it is worth a line, while a decision stands from the second session, because a count is a number an agent can do nothing with. A program this machine has never had is named from two sightings, and a promoted note says it is the project's standing decision rather than this file's. --plain prints the block without the hook envelope, for hosts that take a string back from a handler rather than reading a hook's stdout; pi and omp use it, and they send a lowercase read because neither has a seam that runs before an edit.
  • hook-tool-after (PostToolUse, matched to the command tools) answers a failed command with the pair already on file: the error seen before and what followed it without failing. Two lines at most, 420 bytes, deduped per agent session, and silent unless the store holds a pair for that error. What counts as a failure comes from the harness's exit status where it reports one, and otherwise from the recorded output, by the same rule deja friction uses.
  • hook-precompact (PreCompact) runs wherever a harness has a compaction event — Claude Code, Codex, Cursor, Qwen, Grok Build, Kimi, OpenClaw, pi, omp and prime-agent — and its first job is everywhere: forget what this session was shown, because compaction throws away the blocks while the list that stops them repeating outlives them. Where the host also hands over the transcript path and its own session id, which today is Claude Code and Codex, it captures bounded structured state from the current transcript in the existing index manifest before compaction. The next hook for the same session and workspace returns a recovery packet once, capped at 4 KiB including its trust frame. It records objectives, conclusions, verification commands, explicit gaps/conflicts, provenance, and repository freshness without requiring an agent checkpoint. Capture never waits for a full index rebuild. See automatic compaction recovery for limits, privacy controls, and the raw-actions-to-first-edit metric.

Ranking

Search intersects postings to find candidates, then scores each with BM25 over the query tokens and multiplies in a few bounded signals — each able to break a tie, none able to outrank plain relevance:

  • decision — a session that reached a conclusion is lifted, and the decision-carrying line (not the query-match line) is what recall shows.
  • outcome — a session whose own text reports it reverted an approach and settled nothing else is damped; one that reverted and then settled keeps the decision lift. A promoted rejected/superseded/stale lifecycle state travels with the session and is surfaced on every hit.
  • worn (reuse) — a session agents keep recalling is lifted on a log2 curve capped at +50%, on the theory that what the machine keeps needing is worth surfacing; the cap keeps popularity below relevance.
  • promoted note — a curated note outranks the raw transcript it was distilled from.
  • freshness — recency decays the score gently so time is a hint, not a filter.

recall_context and the hooks reuse the same scored order; the point-of-action decision line comes from the same conclusion extraction the digest uses.

Semantic sidecar

deja embed writes <index-dir>.vectors.bin. The file begins with DJV1, a version, vector dimension, model name, manifest generation, vector count, and covered-record watermark. Each entry stores the records.bin byte offset, its session key, and fixed-width float32 values. Writes use a temporary file and rename. A changed manifest generation or model discards old entries; a corrupt sidecar is treated as absent and rebuilt.

Hybrid reranking

Search first produces lexical BM25 results. When a matching sidecar exists, up to 64 candidates are reranked using the query vector. The final score is 0.5 * normalized lexical score + 0.5 * cosine similarity. A failed query embedding prints one notice and returns the original lexical order.

Add a new harness

Implement the same shape as the existing sources.

Interface:

func LoadNewHarness() []model.Session
func ParseNewHarnessFile(path string) ([]model.Session, error)
func ParseNewHarnessFileFromOffset(path string, offset int64) ([]model.Session, error) // if append-only

Five steps:

  1. Add parser code in internal/sources that returns model.Session with stable Harness, ID, Project, Path, Started, Updated, and Messages.
  2. Add file discovery to currentFiles and harness detection to harnessForPath in internal/index/index.go.
  3. Add load and incremental parse paths in load, parseChangedFile, and parseAppendedFile.
  4. Add install/uninstall config handling in cmd/deja/install.go if the harness supports MCP.
  5. Add fixtures and tests for parsing, indexing, search, and install behavior.