MosaicMemoryCompress

Version: v1.1.0 | Status: Stable | Last updated: 2026-08-17

---

1. Problem

In multi-turn LLM conversations, the context window grows linearly with each exchange. Traditional solutions:

• Session management: Force users to start "new conversations" → lose historical detail, high cognitive burden
• Sliding window truncation: Keep only the last N turns → early critical information lost
• Summary compression: Compress all history into one blob → dialogue structure destroyed, details untraceable

All three require users to understand and manage the concept of a "Session." For non-technical users (writers, creators, etc.), this is unnecessary friction.

---

2. Core Idea

Make Session invisible to users. Simulate human "natural forgetting curve" to enable a logically endless conversation.

Human memory is not all-or-nothing. Recent events are remembered clearly; older events become fuzzy — but important events leave lasting impressions.

MosaicMemoryCompress simulates this process — light compression is structural (rule-based, zero LLM), heavy compression delegates the summarizing judgment to an LLM:

• Recent dialogue: Keep full original text (recent interactions need the most detail)
• Slightly older: Keep message structure, distill content ("de-watering") — Light Compress
• Much older: Merge multiple rounds into a narrative summary — Heavy Compress

The context window never overflows. The user never perceives the existence of a "Session."

---

3. How It Works

MosaicMemoryCompress is a pure, stateless function. Given a message array, it partitions it into three zones by recency:

Message array (R rounds total, from oldest to newest):

Round 1 ────→ Round (R-heavyStart)     │ Heavy zone → ALL → 2 msgs
Round (R-heavyStart+1) → (R-lightStart) │ Light zone → structural truncation, count unchanged
Round (R-lightStart+1) ──→ Round R      │ Raw zone  → keep as-is

Default boundaries (2026-09-05): lightStart=10, heavyStart=40 — 10 vivid
rounds, 20 dehydrating rounds, everything older folded.

Anti-jitter: Compression only fires when R % lightWindow == 0 (Light) or R % heavyWindow == 0 (Heavy). Light compression is pure structural truncation (milliseconds, zero LLM); only Heavy folds make one LLM summary call (~1-2s) at heavy-window boundaries.

Light Compress

Per-message structural truncation — zero LLM calls (since 2026-08-16, data-driven: real-surface token composition is reasoning 33% + tool-call arguments 33% + tool results 24% vs. ~5% text; structural truncation of the big structured payloads yields ~46% net surface savings vs 5.6% from 254 LLM calls):

• reasoning_content: head+tail 30 chars (field preserved; DeepSeek replays it on tool-call turns — truncation API-verified)
• tool_calls arguments: JSON shell preserved, string fields truncated to 120
• tool results: text head 30 + tail 30
• user/assistant text: untouched — stays fresh for the Heavy fold

Incremental: distilled messages carry a _distilled marker; re-triggers skip them.

Heavy Compress

Compress the entire Heavy zone into exactly 1 message — a checkpoint (matching the official backend, which writes a single user checkpoint; no assistant companion, since 0.1.5 requires assistant events to carry settlement fields).

Before (default config, e.g. 70 rounds, the oldest 30 already dewatered):
  [many messages spanning early worldbuilding, character decisions, plot discussions...]

After (2 messages):
  user:      "[Summary] 1) Worldbuilding: soft magic system established
              2) Characters: fall-arc protagonist, female lead, quiet and meticulous
              3) Narrative: fast-paced, subtext-rich, open endings
              4) TODO: supplement M1 commitment list item 5"
  assistant: "[Confirmed] Directions recorded: soft magic, fall arc, fast pacing, M1 TODO."

---

4. Configuration

interface MosaicMemoryConfig {
  lightStart: number;   // Rounds to keep raw. Default 10
  lightWindow: number;  // Anti-jitter for Light. Default 30 (aligned with heavy)
  heavyStart: number;   // Rounds before this enter Heavy zone. Default 40
  heavyWindow: number;  // Anti-jitter for Heavy. Default 30
  callLLM?: (systemPrompt: string, userInput: string) => Promise<string>; // Heavy only; omit for light-only usage
}

---

5. Steady-State Message Count

With default parameters (lightStart=10, lightWindow=30, heavyStart=40, heavyWindow=30):

Heavy zone: 2 msgs   (1 user summary + 1 assistant confirmation)
Light zone: 40 msgs  (20 rounds × 2 — rounds R-heavyStart .. R-lightStart)
Raw zone:   20 msgs  (10 rounds × 2 — the most recent lightStart rounds)
─────────────────
Total:      62 msgs (31 user rounds) (+ 1 system prompt if present)

This count is CONSTANT regardless of how many rounds the conversation has. Whether at round 60 or round 15,000, the message count is always 62 (31 user rounds) for pure two-message rounds — in general 2 + heavyStart × messagesPerRound. The exact value depends on the average number of messages per round (tool-call rounds add an assistant + tool message), but it never grows with R.

---

Measured performance and information-retention numbers live in benchmark/README.md (deterministic simulation + real-LLM spot check). This design document intentionally does not repeat numbers.

6. Design Philosophy

1. Data-driven light, LLM-judged heavy: the light zone is structural truncation of the big structured payloads (reasoning/arguments/results — measured to be ~90% of surface tokens), preserving text verbatim; the heavy zone delegates summarization judgment to the LLM
2. Preserve message skeleton: Light Compress keeps the user/assistant alternation structure — causal chains remain traceable
3. Natural forgetting, not violent truncation: Older = fuzzier, newer = clearer
4. Zero user awareness: No Session concept to understand, no context window to manage
5. Stateless & idempotent: Same input always yields same output, regardless of call history

---

7. Usage

import { mosaicMemoryCompress, type MosaicMemoryConfig, type Message } from 'mosaic-memory-compress';

const config: MosaicMemoryConfig = {
  lightStart: 10,
  lightWindow: 30,
  heavyStart: 40,
  heavyWindow: 30,
  callLLM: async (systemPrompt, userInput) => {
    // Wire to your own LLM provider
    const response = await yourLLM.chat({
      messages: [
        { role: 'system', content: systemPrompt },
        { role: 'user', content: userInput },
      ],
    });
    return response.content;
  },
};

// Call on every user message
const messages = loadConversationHistory();
const compressed = await mosaicMemoryCompress(messages, config);
// compressed is now ready to pass to your main LLM

---

8. Formal Model and the Engineering Choice

This section is the theoretical foundation of the algorithm, not an implementation plan. It explains why MosaicMemoryCompress uses two levels (Light + Heavy) and why Heavy's recursive merge is mathematically sound. The exact multi-level model described below is deliberately NOT implemented — see 10.3 for the engineering rationale.

8.1 The exact model: position is age, nodes compress

Treat the message array as a sequence of memory units:

• Each user node is one memory unit (tool rounds add no user node, so the invariant holds).
• Position is age: counting from the tail (newest) toward the head (oldest), deeper position = more ancient.
• Compression fires once per anti-jitter window (window, default 10):
  • Each window boundary rolls exactly window fresh nodes into the compressible region,
  • so any granularity g ≤ window completes within one window (10 nodes at 2-to-1 → 5 nodes; 5-to-1 → 2 nodes; 10-to-1 → 1 node),
  • and granularities above window have no independent meaning — "merge more into one" is always achievable by repeating "window-to-1".
• The deepest level is incremental Heavy: each window merges the oldest window nodes into a summary of constant size; the summary never grows, the array oscillates and converges to a bound it never exceeds (a calculus-style limit).

Granularity tiers map to human memory:

Memory stage	Human analogue	Granularity
seconds-minutes ago	crystal clear	raw (unchanged)
hours ago	growing fuzzy	g=1 (per-node dehydrate)
days-weeks ago	details lost	g=2 / g=5 (merge nodes)
months ago	only key points	g=10 / g=20
years ago	mere impressions	incremental Heavy (never grows)

The human brain is finite yet never "fills up" — the aged perception that "time accelerates" is exactly the felt experience of old memories being continuously compressed. This model is a discrete simulation of that.

8.2 V1's two levels = the 2-tier special case

• Light (g=1) = tier one of the exact model: per-node dehydration, node count unchanged.
• Heavy (g=∞) = the incremental-Heavy implementation: recursive merge (summaries of summaries); each window merges only the newly rolled-in heavyWindow rounds into the previous summary and always outputs 2 messages — the "window-to-1 incremental" of 10.1, with heavyWindow substituted for the window.

V1's steady-state derivations (constant message count and token size) are therefore direct consequences of the exact model.

8.3 Engineering choice: why two levels (Occam's razor)

Real human-AI conversation round counts:

Scenario	Rounds	Two-level coverage (default 10/40/30/30)
Simple errand	3-5	no trigger (<40), zero cost
Complex feature	20-40	light-only (dewatered from R=40)
Large project	40-70	Light first at R=40, Heavy fold at R=70
Thousand-round dialogue	does not exist	the multi-level model's only target

The multi-level model (g = 1,2,5,10,20…) would pay — for a scenario that does not occur — with higher implementation complexity and more LLM compression calls (each merge is a model call; cost grows with tier count). Benefit does not justify cost. Two levels (V1) are the cost/effect balance: full coverage of the real distribution, minimal complexity, fewest compression calls.

Conclusion: the exact model is the mathematical basis of V1; V1 is the right engineering choice in the real distribution. If thousand-round dialogues ever become real, the multi-level model remains available as a configuration-level extension (a granularity table) without touching the algorithm — but it is not implemented today.

8.4 Future: progressive forgetting tiers (theory, not implemented)

Structural truncation makes the light pass zero-cost, so the two-zone simplification is no longer forced by LLM-call economics — it is a running choice. A finer position-is-age ladder is available whenever needed, e.g.:

• rounds 20–30: keep first/last 200 lines of structural payloads
• rounds 30–40: keep first/last 100 lines
• rounds 40–50: keep first/last 30 lines

Each tier forgets more of the structured payload (reasoning, arguments, results, injections) — content the model's reasoning progressively stops using. User/assistant text is never compressed in any tier, so the conversation's core meaning always survives; the Heavy zone then folds each ancient region into a distilled-forever kernel, keeping the conversation endlessly continuable.

V1 ships the two-zone scheme (light + heavy) and observes real-world effect before any finer tiering.

8.5 Cost model: the cache-breakpoint tax (measured 2026-08-16, refined 2026-09-06)

On providers with automatic prefix caching (DeepSeek, OpenAI), ANY in-place edit of sent history breaks the cache prefix — the first edited node onward is a full miss (DeepSeek prices misses 30× hits). Measured on a real DSH session: 99.7% → 4.2% hit rate on the compression request, recovering to 99.9% immediately after.

The tax is per-window, not per-message: one full-miss request every N rounds, amortized ≈ surface×30/N per round. Early extrapolation (2026-08-16, single-shot fold era) suggested "N=10 ≈ 10× conversation cost"; field measurement under the aligned 10/40/30/30 config (2026-09-06, below) shows the per-window taxes ($0.02–0.10) are repaid 2–10× within the same window — the multipliers below are superseded by the measurements that follow. (Zero-tax alternative: reset-moment enhancement — see ROADMAP M5.)

Field measurement (2026-09-06, live DSH session 85cd44e7, light pass on a 55-round workflow conversation). The heavy-fold numbers above describe the ancient-zone fold; the LIGHT pass behaves differently and far cheaper:

• 706 mid-surface nodes replaced 1:1 (30 rounds dewatered) — surface dropped 553K → 313K tokens (context usage 55% → 39%)
• The cache break is NOT total: prefix caching survives up to the first replaced node (~304K tokens still cache-hit on the next request, 97% of the new surface); the miss is only the replaced span and beyond: 79.8K tokens (79,986 − 221 baseline miss)
• One-time light tax at DeepSeek pricing: ≈ $0.015–0.04 (miss $0.28–0.56/M, hit 10–30× cheaper)
• Payback within the same 30-round window: each following request bills ~240K fewer surface tokens ≈ $0.0067/round saved → ≈ $0.20 per window — ~10× the tax. The dewatered surface pays for the miss within a handful of rounds.

So the earlier ~1.8×-baseline estimate (165-round simulation, heavyStart=30, single-shot fold) overstates the steady-state cost of the aligned design: the light tax is one order of magnitude smaller than the surface savings it buys, and the heavy fold happens once per 30 rounds with a bounded, already-dewatered input.

Heavy fold field measurement (2026-09-06, same live session, R reached the 70-round fold). The heavy fold replaces the oldest 30 rounds with one checkpoint message at the surface head — the next request shows a FULL cache break (cacheRead 490,752 → 2,688; miss 193 → 165,287 tokens ≈ $0.046 at $0.28/M), plus the fold's own LLM call (~10s stall; cost not metered in the event stream, estimated ≈ $0.05 for ~100-250K input tokens). Surface dropped 491K → 168K (−66%; context usage 49% → ~25%). The second request after the fold is already back to 99.6% cache hits.

Structural note: a head-fold always breaks the whole prefix. Requests are time-ordered (oldest first) because that layout maximizes prefix stability (new content lands at the tail); the fold target is by definition the oldest content = the head; prefix caches match from the first token — so replacing the head cannot keep any prefix. This is structural (three constraints: time order / oldest-first compression / prefix-from-head), not a layout bug. The mirror-image layout (newest first) would break the prefix on every new message instead. Keeping the fold input small (pre-dewatered) bounds the LLM call; the head tax itself is unavoidable and is amortized: fold tax + LLM ≈ $0.10 per 30-round window vs ≈ $0.27 of surface savings across the same window (each of ~30 following requests bills ~322K fewer tokens) → net ≈ +$0.15-0.17 per window, cost DOWN. No "2× cost" regime was observed in production; the earlier theoretical multipliers described the heavyStart=30 single-shot era and are superseded by these field numbers.

9. Empirical Case Study: One Event, Three Memory Carriers

A real experiment, anonymized (participants and topic are not reproduced); only the methodological conclusions are kept. The setup is reproducible — any "multi-round design discussion + two compression schemes" combination yields the same comparison.

The event. Two participants — a human designer and an AI assistant — spent several rounds discussing compression-granularity design: merge ratios (2-to-1, 5-to-1, 10-to-1), how each tier maps to human memory stages, and the engineering conclusion to ship two levels while keeping finer tiers as a configuration-level extension. The outcome was written into the design docs.

The treatment. A few rounds later, the same conversation was compressed two ways:

• Archival (industry default): one threshold-triggered full summary, leaving only a structured brief;
• Mnemonic (this library, as configured at experiment time — the earlier 30-round raw window): recent rounds verbatim, older rounds distilled per message, ancient rounds merged into a constant-size summary.

The results.

Carrier	Memory of the event	Form
Human designer	Tiers, mappings, process and conclusion — vivid and accurate	recent memory is verbatim by nature
AI after archival compression	one sentence only ("multi-tier exists but is deliberately not implemented")	tiers, process and data lost; design docs required to recover — "a new person who read a note"
AI after mnemonic compression	a dozen messages kept verbatim, including the designer's own words and experiment data	the discussion sat in the raw zone, never touched; no external reading needed

Conclusions.

1. Humans remember recent events verbatim and keep only lessons and rules from ancient ones — direct behavioral evidence for the tier table in §8.1.
2. Archival compression preserves semantics (rules/conclusions) but loses events (process/detail), and the loss is invisible — the model cannot know what it has forgotten, nor compensate for it.
3. Mnemonic compression matches human behavior because the recent window is never touched: the AI can restate the discussion from context alone.
4. "Never compress the recent N rounds" is therefore not conservatism but the true source of conversational continuity — continuity rests on recent fidelity, not on the quality of any single summary.

10. License

MIT