sleeping-agent

Research code for Salience-Weighted Consolidation (SWC) and baseline context policies on the LoCoMo conversational-memory benchmark (“The Sleeping Agent” paper). Implements gist-style compression with optional temporal-anchor protection, LLM judging, and small aggregate metrics for reproducibility.

Data: LoCoMo is CC BY-NC 4.0 and is not included in this repository. Obtain the benchmark from the upstream release and place locomo10.json (or your export) under data/ locally. Do not commit data/.

Public aggregate results (tracked): results/matched_comparison/matched_stats.json, results/bootstrap_ci/ci_report_10conv_matched.json, results/temporal_preservation/preservation_report_extended.json, results/judge_validation/validation_report.json. Session caches (results/cache/rem/) and raw logs must not be published.

Repository: github.com/kyrkewood/sleeping-agent
Licensed under the MIT license — see LICENSE.

Setup

python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
export ANTHROPIC_API_KEY=...   # or use a .env file (loaded by run_experiment.py)

Run

# Oracle sanity: first QA of conv 0 with ENTIRE dialogue as context (prompt/scoring check)
python run_experiment.py --sanity_check_only

# Debug low F1: first 5 questions — prints context vs evidence, raw prompt, answer, GT, F1 (stderr)
python run_experiment.py --condition truncation --max_conversations 1 --debug_qa 5

# Smoke: first two dialogs, text-only QA; paper-style conditions + full context (parallel)
python run_experiment.py --all --max_conversations 2 --skip_multimodal --parallel

# SWC REM loop sequentially with full log (avoid parallel while diagnosing failures)
./scripts/run_dream_full_capture.sh results/swc_rem_run.log -- --max_conversations 2 --skip_multimodal

# Single condition (--condition truncation | sliding_window | swc_full | swc_temporal | full_context | dream_full | ...)
python run_experiment.py --condition swc_full --conversation 0

# LightMem baseline placeholder (upstream baseline run separately)
python run_experiment.py --write_lightmem_stub

Note: Historic CLI identifiers such as dream_full remain for compatibility with logs and checkpoints; SWC-facing conditions are exposed as swc_full, swc_temporal, etc.

Metrics: token F1 plus Haiku LLM-judge accuracy (mean_llm_judge_accuracy in JSON). Context is full dialogue per conversation (LoCoMo-style scoring). Run JSON separates QA (Sonnet) tokens, REM / consolidation (Haiku) tokens, judge tokens, and build_history wall time — see ASSUMPTIONS.md. Use --no_llm_judge to reduce cost.

Sampling note: Older paper-linked runs sometimes used Anthropic API default sampling. Current code fixes temperature=0 on Anthropic chat calls (src/llm.py::anthropic_text) for reproducible reruns.

See ASSUMPTIONS.md for protocol detail, judge text, multimodal filtering, and LightMem stubs.