Hybrid Episodic Memory

August 7, 2026 · View on GitHub

Agent Memory Challenge · Textual Memory track · Academic division

A memory service implementing the Agent Memory Leaderboard's Add / Search contract. Writes are synchronous. Retrieval fuses a lexical and a dense channel over the original conversational turns with weighted reciprocal rank fusion.

The system is fully deterministic: no LLM is called on either path, so a re-run reproduces byte-identical retrieval.

Submitted configuration

This is what the image runs by default and what the reported score describes.


Lexical channel	BM25, implemented in `src/index.py` — no retrieval library
Dense channel	`BAAI/bge-small-en-v1.5`, frozen, CPU, baked into the image
Fusion	Weighted reciprocal rank fusion, damping `k=60`
Generative LLM	None, anywhere
API keys required	None
Network at runtime	None

Models used in the Add / Search path

Declared explicitly, per the Academic division's model rule:

Role	Model	Notes
Sentence embedding	`BAAI/bge-small-en-v1.5`	Off-the-shelf, not fine-tuned, no training on evaluation data

That is the complete list. No generative language model is invoked in Add or Search. We read the gpt-4o-mini requirement as constraining the generative model in the memory path — preventing capability arbitrage against the platform's fixed answer and judge models — and this submission simply does not use a generative model at all. The one model present is a 33M-parameter frozen encoder used solely for vector similarity.

Should the rule be intended to cover encoders as well, EMBED_ENABLED=false runs the service on BM25 alone, with zero models in the path and no code change.

Method disclosure

What is ours

All of it, in the submitted configuration. Every file under src/ is original code written for this challenge.

Component	File	Description
Contract layer	`src/models.py`, `src/app.py`	`Add` / `Search` / `/health`, including the synchronous-write and exact-ID-echo guarantees
Memory-unit composition	`src/chunker.py`	Same-role and short-message merging, sentence-boundary splitting, timestamp + role baked into the text the answer model reads
Retrieval core	`src/index.py`	Incremental BM25 from the standard formulation, dense cosine, weighted RRF, recency nudge
Fusion and partitioning	`src/store.py`	Per-`user_id` isolation, multi-channel fusion, cross-channel dedup

Prior techniques used, with attribution

Technique	Source
BM25 (`k1=1.5`, `b=0.75`)	Robertson & Walker, Some simple effective approximations to the 2-Poisson model for probabilistic weighted retrieval, SIGIR 1994. Implemented directly; no library.
Reciprocal Rank Fusion	Cormack, Clarke & Buettcher, Reciprocal Rank Fusion outperforms Condorcet and individual rank learning methods, SIGIR 2009. Weighted variant, damping `k=60`.
Sentence embeddings	Xiao et al., C-Pack: Packaged Resources To Advance General Chinese Embedding — the BGE model family. Used as a frozen encoder.

Hybrid lexical+dense retrieval with rank fusion is standard practice and is not claimed as novel. What is specific to this submission is described under docs/DESIGN.md: the memory-unit composition that carries each unit's timestamp and role into the text handed to the answer model, the separate option-augmented retrieval channel for multiple-choice items, and the rank-space recency nudge.

Third-party code present but NOT ENABLED

src/mem0_backend.py and requirements-mem0.txt add an optional LLM fact-extraction channel built on mem0, fused by the same RRF layer.

It is disabled in this submission (MEM0_ENABLED=false) and contributes nothing to the reported score. It is included for transparency and for a future cycle. It has unit tests with mem0 stubbed, but has never been validated end-to-end against a live endpoint.

Disclosed regardless, since the code is in the repository:


Original authors	Prateek Chhikara, Dev Khant, Saket Aryan, Taranjeet Singh, Deshraj Yadav
Technical report	Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory, arXiv:2504.19413 (2025)
Repository	https://github.com/mem0ai/mem0
License	Apache-2.0
Modifications	None. Imported and configured as a library; never patched or vendored.
If enabled	Adds `gpt-4o-mini` (temperature 0) for extraction and `text-embedding-3-small` for its embeddings, and requires `OPENAI_API_KEY`.

No evaluation question, gold answer, or dataset name is special-cased anywhere in this repository.

Quick start

docker build -t agent-memory-challenge .
docker run --rm -p 8000:8000 -e API_KEY=<memory-system-key> agent-memory-challenge

No credentials for third-party services, no network access at runtime, no external dependencies. API_KEY is the shared secret for calling this service.

Method	Path	Auth
`GET`	`/health`	none
`POST`	`/add`	`API_KEY`, via any accepted scheme
`POST`	`/search`	`API_KEY`, via any accepted scheme

/health is unauthenticated and on the same origin as /add, which is what the platform probes when no separate health URL is configured.

Authentication

The default AUTH_MODE=auto accepts the secret through any of the schemes the specification allows, so whichever one the caller uses will work:

Authorization: Bearer <API_KEY>
Authorization: Token  <API_KEY>
X-Api-Key: <API_KEY>

This is deliberate. On the code-submission route the platform chooses the scheme, and pinning a single one in advance would turn a scheme mismatch into a 401 — a status the platform does not retry.

To require one exact scheme instead, set AUTH_MODE to bearer, token or x-api-key. AUTH_MODE=none disables authentication entirely; the specification permits that for public smoke only, so it is not the default. With auto and no API_KEY set, the service runs open — convenient for local development.

Without Docker

python3.12 -m venv .venv && . .venv/bin/activate
pip install --extra-index-url https://download.pytorch.org/whl/cpu -r requirements.txt
API_KEY=<memory-system-key> uvicorn src.app:app --host 0.0.0.0 --port 8000

API

`POST /add` — synchronous

// request
{
  "request_id": "eval:run_abc123:locomo_refined:conv-0:chunk-0",
  "messages": [{ "role": "user", "timestamp": 1704067200000, "content": "memory text" }],
  "user_id": "eval:run_abc123:locomo:conv-0",
  "session_id": "eval:run_abc123:sample:0"
}
// response — 200 only once the memory is queryable
{ "success": true, "request_id": "…", "user_id": "…", "session_id": "…" }

Indexing and embedding both complete before the response, so a 200 means the next Search can see the memory. Repeated request_ids are idempotent, because Add is retried on 408/409/425/429/5xx.

`POST /search`

// request — options present only on multiple-choice items
{ "query": "…", "options": ["A. …", "B. …"], "user_id": "…", "top_k": 100 }
// response
{ "data": [{ "id": "mem_0", "content": "[2024-01-01 00:00] user: …", "score": 0.87,
             "created_at": "2024-01-01T00:00:00Z" }] }

Results are ordered best-first — the platform preserves our order and reads at most top_k. user_id is a hard partition; memories never cross it.

Configuration

Every knob is an environment variable, so the published image runs unmodified.

Variable	Default	Purpose
`API_KEY`	—	Shared secret for calling this service. Unset + `auto` runs open
`AUTH_MODE`	`auto`	`auto` accepts Bearer / Token / X-Api-Key; or pin one, or `none`
`MAX_RETURN`	`100`	Cap on returned memories. Primary tuning knob — see below
`EMBED_ENABLED`	`true`	`false` → BM25 only, no models at all
`EMBED_MODEL`	`BAAI/bge-small-en-v1.5`	Baked into the image at build time
`CANDIDATE_DEPTH`	`400`	Per-channel depth before fusion
`RRF_K`	`60`	Rank-fusion damping
`OPTION_CHANNEL_WEIGHT`	`0.6`	Weight of the option-augmented query channel
`RECENCY_WEIGHT`	`0.15`	Recency nudge in rank space; `0` disables
`MIN_UNIT_CHARS` / `MAX_UNIT_CHARS`	`24` / `1200`	Merge-forward and split thresholds
`BM25_K1` / `BM25_B`	`1.5` / `0.75`	BM25 saturation and length normalisation
`MEM0_ENABLED`	`false`	Optional extraction channel — off in this submission

On `MAX_RETURN`

top_k is a ceiling, not a requirement, so how many memories to return is ours to choose — and it cuts both ways. The platform's judge penalises over-answering on list questions (gold A, B, C versus generated A, B, C, D scores WRONG), which argues for fewer, higher-precision memories. Recall on the large-context datasets argues for more. The default returns the full top_k; it is the first thing to sweep once private-set feedback exists.

Tests

python -m pytest tests/ -q                              # 42 assertions
uvicorn src.app:app --port 8000 &                       # then end-to-end:
python scripts/smoke_local.py --base-url http://127.0.0.1:8000

Suite	Covers
`tests/test_contract.py`	Every rule the platform enforces by failing a stage outright, including the ones that bite under HTTP 200
`tests/test_concurrency.py`	64 concurrent `Add` and 32 concurrent `Search` — no lost writes, no duplicate ids, idempotent retries
`tests/test_fusion.py`	The multi-channel fusion layer, with the optional extraction channel stubbed

The contract suite is deliberately paranoid because a full evaluation is allowed once per three months: success must be boolean true, all three IDs must echo byte-for-byte, data must be an object field rather than a bare array, and every item needs a non-empty id and content. Each of those fails the stage under HTTP 200 if wrong.

CI runs the suite and builds the Docker image on every push.

Layout

src/models.py         contract request/response models
src/chunker.py        messages -> indexable units (timestamp + role in the text)
src/index.py          incremental BM25, dense cosine, RRF, recency
src/embedder.py       frozen bge encoder with lexical fallback
src/store.py          per-user_id partitioning and channel fusion
src/app.py            POST /add, POST /search, GET /health
src/mem0_backend.py   optional extraction channel — disabled in this submission
docs/DESIGN.md        why each design decision is what it is
scripts/smoke_local.py  end-to-end check against a running instance

The platform's public evaluation code (AML-memory/agent-memory-leaderboard, commit 34405461) informed several decisions documented in docs/DESIGN.md. It is referenced rather than vendored here, since that repository carries no license.

Data handling

Evaluation data and derivatives are used solely to serve the current task. Nothing is used for model training, fine-tuning, product analytics, dataset reconstruction, or redistribution. Memories are held in process only — no database, no disk persistence, no request-body logging — so stopping the container discards everything, well inside the 30-day deletion requirement.

License

MIT. See LICENSE.