Design and decisions

August 25, 2026 · View on GitHub

Scope

dsh-memory is a DeepSeek Harness bundle: a Host pipeline plugin plus a Web settings page. It turns historical root-session logs into durable long-term memory — a two-phase extraction and consolidation pipeline adapted from the Codex memories system onto DSH primitives, plus a read path that injects a dense summary into every session and exposes four retrieval tools.

The plugin owns its memory directory and its KV-backed bookkeeping. It never rewrites session logs, never touches the agent preset or the prompt template, and runs entirely asynchronously after session boundaries.

Decision record

Confirmed with the user before implementation (2026-02):

Decision	Choice	Rationale
Memory root	`$DSH_HOME/memories` (configurable)	Codex-style global consolidation; workspace context is preserved per rollout instead
Phase 2 executor	Direct `llm.stream` call	Lightweight, retryable, no subagent quota cost; consolidation is one bounded prompt
Read path	Summary injected + tools on demand	`memory_summary.md` is token-budgeted navigation; details stay in MEMORY.md / rollout summaries
Trigger	Startup catch-up + turn-end debounce	Codex startup task plus DSH event-driven freshness; idle debounce doubles as the "idle long enough" guard

Confirmed with the user before implementation (2026-08):

Decision	Choice	Rationale
Skill promotion	Phase 2 emits `skills/<name>/SKILL.md` via optional tool args	Codex skills parity; strictly validated names/paths; `remove_skills` for explicit retirement
Forgetting	Usage-tracked candidates retired into `rollout_summaries/.retired/`	Codex `max_unused_days` parity; recoverable archive instead of hard delete
Usage source	`memory_read` / `memory_search` calls (not injection, not list)	Mirrors Codex read-path telemetry: only explicit retrieval counts as "used"
Run logs	`logs/<stamp>-<stage>.json`, newest 50 kept	Settings page "what did the AI do" panel; bounded storage, redacted content
Running state	`get-state` exposes phase1/phase2 single-flight flags; buttons disabled while running	Reopening the settings page no longer "unlocks" an in-flight run
Per-phase reasoning effort	`phase1ReasoningEffort` / `phase2ReasoningEffort` (empty = provider default), validated against adapter metadata per run	Users trade latency/cost vs extraction/consolidation depth independently; unsupported values degrade to the default instead of failing calls

Architecture

session logs ──(sessionQuery/sessionPersistence)──► Phase 1 (per session)
                                                      ├─ filter → render → redact
                                                      ├─ llm extract → {raw_memory, rollout_summary, rollout_slug}
                                                      ├─ write rollout_summaries/<slug>.md
                                                      ├─ append raw_memories.md
                                                      └─ write run log (per session outcomes)
                                                                  │
                                                    (cooldown, pending flag)
                                                                  ▼
                                          Phase 2 (global, single-flight)
                                                      ├─ llm consolidate (INIT / INCREMENTAL)
                                                      ├─ write MEMORY.md + memory_summary.md (v1, redacted)
                                                      ├─ promote/remove skills/ (validated)
                                                      ├─ retire candidate summaries → .retired/
                                                      ├─ rotate raw_memories.md → archive
                                                      └─ write run log (full model output)
                                                                  │
                                                                  ▼
                                          read path: systemPrompt section (summary, cached)
                                                     memory_list / memory_read / memory_search / memory_add
                                                     (read/search record usage → forgetting input)

Codex concepts → DSH primitives

Codex	dsh-memory
state DB (`threads`, `stage1_outputs`, `jobs`)	storage hub KV (`json` backend): claims, cooldown, pending flag, overrides, usage, retiredAt
rollout JSONL files	session logs via `sessionQuery.readSession` / `sessionPersistence.load`
startup task, stage-1 job leases, retry backoff	boot catch-up + `agent/turn-stopping` debounce; per-session claims with backoff and restart recovery
global Phase 2 lock + 6h cooldown	in-process single-flight + configurable cooldown (default 6h)
`~/.codex/memories` + git baseline	`$DSH_HOME/memories`; atomic writes (temp + rename); no git dependency
developer-policy injection + `list`/`read`/`search`/`add_ad_hoc_note`	`systemPrompt.section` with a per-assembly provider + `memory_list`/`memory_read`/`memory_search`/`memory_add`
secret redaction, no-op gate, `v1` first-line protocol	kept: redaction on both input and output, empty-field no-op, exact `v1` first line
read-path usage telemetry (`usage.rs`) + `max_unused_days`	`memory_read`/`memory_search` usage records + `maxUnusedDays` candidates + `.retired/` archive
skills promotion in consolidation prompt	`PHASE2_TOOL` optional `skills`/`remove_skills` args, validated writes

Storage layout

<root>/
  memory_summary.md          always-injected navigation layer (first line exactly `v1`)
  MEMORY.md                  grep-friendly handbook: preferences, procedures, failure shields
  raw_memories.md            Phase 1 output awaiting consolidation
  raw_memories.archive.md    rotated history of consolidated raw blocks
  rollout_summaries/<slug>.md  per-session recaps with session id / cwd provenance
  rollout_summaries/.retired/  forgotten recaps (recoverable archive, never deleted)
  skills/<name>/SKILL.md     procedures promoted by consolidation (+ scripts/, templates/, examples/)
  logs/<stamp>-<stage>.json  run logs: every phase1 batch and phase2 call with full model output
  extensions/ad_hoc/notes/   user-requested ad hoc notes (memory_add)

Bookkeeping

One KV record holds { processed: sessionId → claim, lastPhase1At, lastPhase2At, phase2Error, pendingConsolidation, overrides, usage: path → {count, lastAt}, retiredAt: slug → timestamp }. Claims are running | done | noop | failed(attempts), so a restart never re-extracts a session, never re-runs consolidation inside the cooldown, and interrupted running claims recover as failed after 30 minutes. Failed claims retry with exponential backoff and give up at retryLimit. Writes are serialized through a promise chain; when the KV backend is unavailable the store degrades to process-local state. usage/retiredAt are optional on load so archives written by older versions keep loading.

Scheduler

Boot: one delayed pipeline run (4s), then once per root agent/session-start (throttled 60s).
Every root agent/turn-stopping: debounced run (default 3 min) — the debounce is the "session idle long enough" gate that prevents summarizing active sessions.
Phase 1 selects root sessions only (subagent sessions excluded), skips processed ones, ages out sessions older than maxRolloutAgeDays, extracts with bounded concurrency, then sets pendingConsolidation. Every batch writes a run log.
Phase 2 runs after any successful extraction, pending flag, or unconsumed ad hoc notes, subject to the cooldown; failures record phase2Error and retry on the next scheduled window or manually. Every attempt (skipped/error/consolidated) writes a run log.
Both runners expose isRunning; get-state reports it so the settings page keeps buttons disabled (and polls every 4 s) while the pipeline is genuinely in flight.

Incremental extraction

Sessions grow after their first extraction, so claims carry a lastSeq watermark. A processed session is re-checked no more often than recheckIntervalMs (default 30 min); when its log grew by at least minDeltaEvents new rendered events, the delta is extracted into a new rollout part (<slug>-partN) and appended to the raw queue. Legacy noop claims without a watermark get one full re-extraction; legacy done claims only get their watermark baselined (their content is already consolidated). The model input for a delta states explicitly that earlier content was already processed.

Structured tool-constrained output

Both phases constrain model output through the native function-calling channel (GenerateOptions.tools): Phase 1 must call memory_save with raw_memory, rollout_summary, rollout_slug; Phase 2 must call memory_write with memory_md and memory_summary_md plus optional skills / remove_skills / retire_summaries arrays. The plugin writes files itself; the model only fills structured arguments. Free-text parsing remains as a fallback for models that ignore tool schemas. Phase 1 also retries once with a halved transcript when output is truncated.

Reasoning effort per phase

Each phase may be given its own reasoning effort (off/low/high/max on DeepSeek; empty uses the provider default). At the start of every run the runner resolves the adapter-owned effort list for the exact route through llm.resolveModelInfo and applies the configured id only when the model actually supports it; unknown values (or metadata failures) degrade to the provider default instead of failing the call, since DSH rejects unsupported efforts before provider I/O. The settings page offers the adapter-reported efforts (falling back to the DeepSeek levels) for each phase, and run logs record the effort actually applied.

Skill promotion

The consolidation prompt instructs the model to promote repeated, verifiable procedures into skills/<name>/SKILL.md (YAML frontmatter + triggers/procedure/verification/pitfalls), with optional supporting files under scripts/, templates/, examples/. Skill names are validated against [a-z0-9][a-z0-9-]{0,79} and auxiliary paths against the allowed subdirectories before any filesystem call; remove_skills deletes only validated skill directories. A failing skill write is recorded (run log skillErrors) without failing the rest of the consolidation. The existing skills/ inventory is fed back into the model so updates merge instead of duplicating. To make promoted skills load into sessions, the agent preset's customSkillDirs must reference the memory skills/ directory; the plugin never touches presets.

Forgetting

Read-path usage is recorded per file when memory_read returns content or memory_search surfaces hits (listing and injection are not usage, mirroring Codex read telemetry). Phase 2 annotates rollout summaries older than maxUnusedDays (default 30, 0 disables) since their last activity as 「候选遗忘」and asks the model to compress/remove their MEMORY.md/summary entries and list the retired slugs in retire_summaries. The plugin moves those files into rollout_summaries/.retired/ (recoverable; retired slugs stop appearing in the phase-2 index). Forgetting is conservative by prompt construction: only explicitly annotated candidates may be retired.

Ad hoc notes

memory_add writes user-requested notes into extensions/ad_hoc/notes/; Phase 2 feeds every unconsumed note into consolidation as the highest-priority evidence source (user self-reports outrank inferred facts, and conflicting evidence is preserved with attribution on both sides). Consumed notes are moved to extensions/ad_hoc/archive/ after a successful consolidation.

Security invariants

Session content is rendered as data, never instructions: the extraction prompt states it, and the renderer strips everything except user/assistant text and tool call/result summaries.
Secret redaction applies to the rendered transcript and to model output before either is written to disk: API keys, JWT, Bearer tokens, PATs, Slack tokens, PEM private keys, and password/secret/token/api_key-style assignments.
Memory tools resolve paths inside the memory root only; traversal and absolute paths are rejected before any filesystem call.
The /dsh-memory/rpc endpoint guards itself because the DSH webServer performs no auth/CSRF checks: requests with a non-loopback Host (DNS rebinding) or a non-loopback Origin (cross-origin browser POSTs, including Origin: null) are refused with 403; requests without an Origin (curl/local tooling) are allowed since such clients already hold machine access.
The vault cannot delete what forgetting only retires: delete-file hard-refuses the core pipeline files (MEMORY.md, memory_summary.md, raw_memories.md, raw_memories.archive.md) and everything under rollout_summaries/.retired/.
Usage telemetry records only reads of active rollout summaries (canonicalized path keys); handbook/log/skill/retired reads never count as "used" and never enter the forgetting input.
The consolidation prompt demands the exact v1 first line and the plugin enforces it mechanically (ensureSummaryV1).
Consolidation output is bounded by phase2MaxTokens; injected summaries by maxSummaryChars; transcripts by maxTranscriptChars; raw input by maxRawChars. Every model input has a cap.

Failure modes and recovery

Failure	Behavior
No model route	Pipeline skips with `skippedNoRoute`; no state damage; retried on next trigger
Extraction call fails	Claim marked failed with backoff; `retryLimit` attempts, then parked
Consolidation output malformed	Raw memories preserved, `phase2Error` recorded, pending flag kept
Consolidation output truncated	`finish_reason=length` inside the tool-call JSON: raised `phase2MaxTokens` (default 32768, cap 131072) + lenient block/JSON fallbacks + redacted model response in the run log; auto-retries pause after 3 consecutive failures (manual runs bypass)
Skill write invalid	That skill is skipped with the error in the run log; MEMORY.md/summary still land
Retire target missing	Skipped silently (ENOENT), no state change
Process restart mid-extraction	Orphaned `running` claims recover as failed and retry
KV backend missing	Claims stay process-local; pipeline still runs without cross-restart durability
Session log unreadable	That session's claim fails; other candidates unaffected

Coverage boundary

The plugin reads session logs through the public query surface and writes only inside its own memory root. It does not modify session history, the agent preset, prompt templates, or other plugins' storage. Extraction runs only for root (non-subagent) sessions; subagent work is already summarized inside its parent's log. Model routing falls back to the deployment default model (agentDefaultModel) when no dedicated route is configured. The client page is a configuration and observation surface only; it does not gate any security boundary.

Prior art

The two-phase pipeline, three-layer artifact layout, no-op gate, secret redaction, claims with retry backoff, cooldown, skill promotion, read-path usage telemetry with a max_unused_days forgetting threshold, and the v1 summary protocol are adapted from the OpenAI Codex memories system (audited via codex-rs/memories, codex-rs/ext/memories, codex-rs/state). Prompts and code are written from scratch for DSH; no Codex implementation text is copied. The package build chain follows the @nanmicoder/dsh-auto-mode layout, and the settings page pattern follows the @dsh-local/vision-bridge plugin.