DiffMem (Git-Native Agent Memory) Evaluation

September 9, 2026 · View on GitHub

Resource: DiffMem, a git-native / file-based memory backend for AI agents Source: github.com/Growth-Kinetics/DiffMem Source analyzed: DiffMem Author: Alex / Growth Kinetics (powers the "Annabelle AI" product) License: claimed MIT (pyproject classifier) but no LICENSE file exists, GitHub detects none Evaluated: 2026-07-25 (code + health + docs, each by a dedicated agent)

Score: 3/5 (Moderate, case study only)

Decision: Integrate as a case study and a concept in guide/core/memory-systems.md §3.7, not as a recommended tool. The documentable idea is LLM-agentic git-shell retrieval (an LLM that greps and blames the repo instead of querying any index), a point in the memory design space the guide did not cover. Do not recommend adoption: bus factor one, no license file, the retrieval core is untested and being rewritten. Carry forward one sharp teaching point (match retrieval to query shape) and two reusable primitives.

The premise flipped: DiffMem deleted BM25

The starting assumption (from a Perplexity survey) was "Markdown + Git + in-memory BM25". That is no longer true of the code. Commit c4fe4a3 ("git based retrieval") removed the entire bm25_indexer/ module (a real rank_bm25.BM25Okapi implementation, 114 lines, self-labeled "PoC minimal") and replaced it with an LLM-driven shell-command agent. The current README states it twice (README:9, README:345): "No vector databases, no embeddings, no BM25, just git and an LLM." The old BM25 is only recoverable via git show c4fe4a3^:src/diffmem/bm25_indexer/indexer.py.

This matters for our own context: DiffMem is not a validation of the file-plus-BM25 approach. They tried it and walked away, toward full-LLM retrieval. For a lexical query shape (error strings, symbols) that is arguably the wrong direction. See the teaching point below.

What It Is (verified against code)

Storage: Markdown files in a git repo, real multi-tenant layout. One central repo holds each user as an orphan branch user/<id>, mounted via git worktree (local_storage.py:83-111). Orphan branches share no history, so isolation is a genuine git guarantee, not a folder convention. Per-user files: <user>.md, memories/{people,contexts}/*.md, timeline/YYYY-MM.md, autogenerated index.md. Frontmatter is YAML parsed with pure PyYAML, no LLM (frontmatter.py:23-56).
Retrieval: no ranking algorithm at all. run_retrieval_agent() (retrieval_agent/agent.py:197-308) runs an LLM tool-calling loop (default 4 turns) with exactly one tool, run(command=...). The model shells out to whitelisted commands (cat, head, tail, grep, ls, wc, git log/diff/blame/show/rev-list) through command_router.py, output capped 150 lines / 30KB, 10s timeout, executed via subprocess.run(shell=True) (command_router.py:265). A deterministic LLM-free baseline loads the user entity plus five recent timeline files (baseline.py, docstring: "No BM25, no embeddings, no LLM calls, pure file reads").
Temporal / differential memory: wired, but LLM-driven. Git history is queryable data the agent pulls on demand (git blame to find when a fact last changed). The consolidator's _link.py:29-56 does programmatic co-change analysis: it walks recent commits to build an entity co-occurrence graph, then an LLM proposes wikilinks. Genuine differential signal, LLM for the write.
Machinery (core vs optional): core is a Python library (DiffMemory in api.py) with 4 dependencies (requests, openai, gitpython, python-dotenv). Around it: writer agent (1276 lines, largest file), consolidator (4 tools: dedupe/link/redistribute/reabsorb, ~1300 lines), a pluggable executor (inline default, hatchet durable-queue optional), an 890-line FastAPI server, pluggable ontologies (personal/corporate). The optional layers are gated behind env vars and requirements-server.txt. The core/server split is honest.

Scoring Breakdown

Criterion	Score	Rationale
Relevance to CC users	3/5	Agent memory, file-based, but OpenRouter/OpenAI-SDK driven, not CC-specific.
Novelty vs. guide	4/5	LLM-agentic git-shell retrieval (no index) is a genuinely new point vs the vector/graph tools in `memory-systems.md`.
Technical quality	3/5	8255 src LOC / 5856 test LOC (0.71 ratio, healthy) and a clean core/server split, but the retrieval agent has zero tests, `cli.py` is declared in `pyproject.toml:62` yet does not exist (broken console script), version drift 0.4.0 vs shipped v0.5.
Evidence quality	2/5	"Production, powers Annabelle" but one workload; "50-year durability" unsupported; no eval harness (the notes' Eval section is four bookmark links). Latency and cost documented honestly though.
Maturity / health	2/5	Bus factor one (Alex 78% under 7 aliases), no LICENSE file, no tags, no releases, not on PyPI. 897 stars is real traction but a classic star-vs-maintainership gap.

Overall: 3/5. More substantial than a weekend project (real production use, 897 stars (now 898 as of 2026-07-28), 8K LOC, strong docs) but carries hard disqualifiers (bus factor one, no license) plus an untested, in-rewrite retrieval core.

The one teaching point worth keeping: match retrieval to query shape

DiffMem built BM25, then deleted it for full-LLM retrieval. That is a real-world signal, and the reason is instructive. DiffMem's queries are natural-language questions about people, events, and relationships (a conversational companion recalling "what did the user say about their sister in March"). That is a semantic query shape, where keyword BM25 underperforms and entity/temporal reasoning matters, so they escalated to an LLM that can git blame and reason.

The inverse holds for a coding-agent retex system: the queries are error strings, stack traces, and code symbols, a lexical query shape where term overlap is high and BM25 is not just sufficient, it is often better than embeddings and far cheaper than an LLM loop. So DiffMem's abandonment of BM25 is not evidence that BM25 is wrong. It is evidence that retrieval must fit the query shape. Same substrate (files + git), opposite retrieval choice, because the queries differ. This is the sharpest, most transferable lesson from the whole analysis and belongs in the guide.

Patterns worth extracting (for the guide and for a retex system)

LLM-agentic git-shell retrieval (command_router.py, retrieval_agent/agent.py). Instead of a search DSL or an index, give the LLM grep + git log/blame/diff behind a base-command whitelist, output truncation, and a timeout. About 351 lines for a very flexible retrieval surface, and git blame-style "when was this last true" comes free. Novel vs the guide. Caveat to document: arg-level validation is missing (only tokens[0] is checked before shell=True), so $(...) inside an argument is a real injection risk if ever exposed to untrusted input.
Orphan-branch-per-user + worktree isolation (local_storage.py:83-111). Real filesystem and history isolation across tenants with one shared object store, cleaner than "one folder per user."
Frontmatter as the query surface, prose as the payload (frontmatter.py). Structured queryable fields in YAML, narrative in the body. Directly maps to a retex file: severity, tags, prevention_rule in frontmatter, story in body.
(Minor) PID-liveness lockfile with staleness reclaim (lock.py) and commit decoupled from remote backup (repo_manager.py:97-117). Good minimal primitives for a file-based memory running as a service.

Explicitly not worth copying: the no-LLM-free mode. Every read and write hits OpenRouter (api-surface.md:9-10), writes take 60-600s, reads 5-30s, at a per-op cost. For a retex system queried constantly during debugging, that latency and cost tax is exactly what a keyword/BM25/frontmatter-tag filter avoids.

Weaknesses

Bus factor one: Alex / Growth Kinetics is 78% of 60 commits under 7 git aliases. If Alex stops, the project stops.
License gap: MIT claimed only in the pyproject classifier, no LICENSE file, GitHub detects no license. A legal blocker for adoption.
Retrieval core untested and in flux: the highest-blast-radius component (an LLM freely generating shell commands) has zero test coverage, and a rebuilding-retrieval branch is open. Do not build on a moving, untested part.
Broken console script: pyproject.toml:62 declares diffmem = "diffmem.cli:main" but cli.py does not exist. pip install -e . && diffmem crashes.
"Production" oversold: production for one workload (Annabelle), the roadmap admits the PoC indexing strategy still needs hardening (too memory-intensive), "zero external dependencies" is contradicted by a mandatory LLM API, "50-year durability" has no supporting data.

Integration Decisions

Item	Decision	Rationale
Document LLM-agentic git-shell retrieval in `memory-systems.md` §3.7	Done	New point in the design space, distinct from vector/graph/BM25. Includes the injection caveat.
Add the "match retrieval to query shape" teaching point	Done	Transferable, sourced from DiffMem's BM25 removal. Sharpest lesson.
Recommend DiffMem as a tool	Skip	Bus factor one, no license, untested in-rewrite retrieval.
Full standalone tool entry in `memory-systems.md` §3	Skip	Catalog-bloat reasoning; it is a case study inside §3.7, not a recommendation.
Note orphan-branch isolation primitive	Done (§4.7)	Useful for multi-user/multi-project file memory.
Entry in `credits.md`	Add	Growth Kinetics, MIT-claimed.

For the retex / smart-suggest-routing angle

DiffMem confirms, by counter-example, that the existing BM25 smart-suggest-routing engine is the right tool for the retex query shape (lexical error strings), and that DiffMem is not a competitor to copy. The reusable idea from DiffMem is the escalation tier: BM25 or a frontmatter-tag filter as the cheap default, and an LLM-agentic git-shell pass only when the lexical match misses. That is the pragmatic hybrid, built on the engine already in production, not on DiffMem's clone-only, LLM-required, single-maintainer stack.