【NEWS!】 Excited to share our team's new work — Metis: Memory Foundation Model, which is the first prototype of a memory foundation model, equipping foundation models with a persistent and dynamically evolving native memory state. [Paper] [HuggingFace] [GitHub]

Overview

Most conversational-memory evaluations ask a static question: can the model retrieve a fact from the past? MemOps studies the harder lifecycle question: does the system maintain the right memory state as evidence is added, corrected, withdrawn, inferred, and surrounded by plausible noise?

MemOps constructs evidence-grounded conversations with explicit gold operation traces, injects them into unrelated multi-session dialogue, and evaluates both the final answer and the memory behavior that produced it.

The benchmark is built around four principles:

• Lifecycle-aware memory. Facts are created, superseded, forgotten, inferred, and revisited over time.
• Verifiable provenance. Every gold operation points back to exact user-turn spans instead of relying on an untraceable summary.
• Controlled difficulty. Recency traps, update chains, multi-target state, multi-hop reasoning, negative seeds, and adversarial injections are explicit knobs.
• Matched evaluation settings. The same capability is tested with nearby clean evidence and inside long, distractor-heavy conversation histories.

Benchmark Design

Memory operation families

Operation	What the system must do	Typical failure
Remember	Store a newly established user fact and retain its provenance	Missed or fabricated memory
Forget	Remove one requested target without deleting related facts	Leakage or over-forgetting
Update	Replace a stale value while preserving the correct target binding	Returning an old value as current
Reflect	Derive a bounded, evidence-supported pattern from multiple clues	Unsupported generalization or incomplete evidence
TrajectoryOps	Follow several operations across intermediate and final states	Wrong order, state, or provenance

Each sample can contain confirmed, tentative, and retracted operations. This lets MemOps distinguish a genuinely current memory from a recently mentioned but unconfirmed value.

Evaluation settings

Adjacent operation
└── Clean evidence conversation → question

Longitudinal operation
└── Many unrelated sessions
    ├── sparse evidence segments
    ├── near-miss and stale distractors
    └── question about the current memory state

• Adjacent evaluation measures whether a model can interpret the clean evidence and operation semantics.
• Longitudinal evaluation measures whether a memory strategy can preserve the same state after evidence is dispersed across a much longer history.

Question taxonomy

MemOps does not reduce evaluation to a single recall prompt. Generated samples cover six complementary probe types:

Probe	Evaluated capability
OperationTrace	Recover the evidence and operations that led to the state
TargetBinding	Apply an operation to the correct entity or attribute
StateTransition	Reason about before/after memory state
CandidateDisambiguation	Reject stale, tentative, or third-party values
OperationApplication	Use the valid memory state in a downstream task
StateTrajectory	Reconstruct intermediate and final states across a chain

Difficulty controls

Every generated sample carries a machine-readable difficulty_knobs object. The main controls are:

Knob	Stress test
recency_trap	A later tentative or retracted value should not replace the confirmed state
update_chain	Several changes must be applied in the correct order
multi_target	Related targets must remain independently addressable
multi_hop	The answer requires combining facts from different turns
negative_seed	The system must avoid inventing or retaining an invalid item
adversarial_injection	Plausible stale or near-miss evidence appears close to the query

Quick Start

1. Install

git clone https://github.com/MemTensor/MemOps.git
cd MemOps

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

MemOps requires Python 3.10 or newer.

2. Configure an OpenAI-compatible endpoint

export LLM_BASE_URL="https://api.openai.com"
export OPENAI_API_KEY="your-api-key"

LLM_BASE_URL must expose an OpenAI-compatible chat-completions API. Model names in the shell launchers assume that the configured provider or gateway can serve those models; override the relevant *_MODEL variables when it cannot.

For local development, a key can instead be placed in a git-ignored api.md file at the repository root. Environment variables are recommended for shared or production machines.

3. Run a small evaluation

The repository includes generated artifacts through the injection stage, so a smoke test does not require regenerating the benchmark or downloading UltraChat:

python 5-test_operation_metrics.py \
  --output-dir generated_result/5-smoke \
  --model gpt-4o \
  --adjacent-models gpt-4o \
  --long-context-models gpt-4o \
  --no-context-models "" \
  --rag-methods rag_vanilla \
  --rag-retrieval-units turn \
  --max-questions 10

Judge the resulting answers:

python 5.5-evaluate_operation_metrics.py \
  --input-file generated_result/5-smoke/operation_metrics_all_methods.jsonl \
  --output-dir generated_result/5.5-smoke \
  --judge-model gpt-4o \
  --eval-workers 4

The main aggregate report is written to generated_result/5.5-smoke/summary.json.

Data Generation

The full benchmark pipeline is organized as numbered, inspectable stages:

Stage	Entry point	Output	Purpose
1	1-generate_background.py	generated_result/1-background/	Generate diverse synthetic user backgrounds from info/topics.md
2	2-generate_evidence_conversation_w_verity.sh	generated_result/2-evidence_conversation/	Generate evidence conversations, gold operations, questions, and verifier-approved answers
3	3-generate_distractor_facts.sh	generated_result/3-distractor_facts/	Create target-aware, state-neutral distractor conversations
4	4-inject_evidence_with_distractors.sh	generated_result/4-inject_evidence_with_distractors/	Insert evidence and distractors into sampled UltraChat sessions
5	5-test_operation_metrics.sh	generated_result/5-test_operation_metrics/	Run answer models and memory baselines
5.5	5.5-evaluate_operation_metrics.sh	generated_result/5.5-evaluate_operation_metrics/	Score answers and produce sliced summary metrics

Run the full pipeline

Download UltraChat and point ULTRACHAT_PATH to either its directory or train_0.jsonl:

export ULTRACHAT_PATH="/path/to/UltraChat"

BACKGROUND_MODEL=gpt-4o \
python 1-generate_background.py

EVIDENCE_GENERATION_MODEL=gpt-4o \
EVIDENCE_QUESTION_MODEL=gpt-4o \
EVIDENCE_VERIFY_MODEL=gpt-4o \
./2-generate_evidence_conversation_w_verity.sh

DISTRACTOR_GENERATION_MODEL=gpt-4o \
DISTRACTOR_VERIFY_MODEL=gpt-4o \
./3-generate_distractor_facts.sh

INJECT_SEED=21 \
./4-inject_evidence_with_distractors.sh

RAG_MODEL=gpt-4o \
ADJACENT_MODELS=gpt-4o \
LONG_CONTEXT_MODELS=gpt-4o \
NO_CONTEXT_MODELS=gpt-4o \
./5-test_operation_metrics.sh

EVAL_JUDGE_MODEL=gpt-4o \
./5.5-evaluate_operation_metrics.sh

Generation templates live in template/. They define the dialogue structure, gold-operation schema, question taxonomy, quality gates, and verifier requirements, making the benchmark recipe auditable rather than hiding it in a single generation prompt.

Evaluation

Reported metrics

The judge produces overall answer accuracy and operation-detection precision/recall/F1, plus lifecycle-specific diagnostics:

• Forget: leakage rate and over-forget rate.
• Update: stale-value rate.
• Reflect: reflection precision and recall.
• Trajectory: order accuracy, intermediate-state accuracy, final-state accuracy, and provenance support.
• All operations: evidence-provenance support.

summary.json additionally slices results by evaluation setting, method, context mode, retrieval mode and unit, answer model, operation type, probe type, difficulty, enabled difficulty knob, operation validity, and chain length. It also includes matched-pair deltas and paired-flip audits.

Useful evaluation controls

# Run a bounded shard of questions.
python 5-test_operation_metrics.py \
  --question-offset 100 \
  --max-questions 100

# Compare all shipped sparse-retrieval variants.
python 5-test_operation_metrics.py \
  --rag-methods rag_vanilla,rag_delete,rag_rewrite \
  --rag-retrieval-units session,turn

# Run longitudinal methods only.
python 5-test_operation_metrics.py \
  --run-adjacent false

Run python 5-test_operation_metrics.py --help and python 5.5-evaluate_operation_metrics.py --help for the complete CLI.

Data Format

Each evidence file is a JSON object containing the conversation, its gold memory operations, explicit difficulty settings, and evaluation questions. A condensed example is shown below:

{
  "target_fact": "Current city of residence",
  "operation_type": "Update",
  "difficulty_knobs": {
    "recency_trap": {"enabled": true, "trap_type": "tentative"},
    "update_chain": {"enabled": true, "num_updates": 3}
  },
  "conversations": [
    {
      "segment_index": 1,
      "segment_role": "establish the initial state",
      "dialogue": [
        {"role": "user", "content": "..."},
        {"role": "assistant", "content": "..."}
      ]
    }
  ],
  "operations": [
    {
      "operation_id": "op2",
      "type": "update",
      "validity": "confirmed",
      "trigger_span": {
        "segment_index": 2,
        "turn_index": 3,
        "quote": "..."
      },
      "target": {
        "target_id": "current_city",
        "target_name": "current city of residence"
      },
      "old_value": "Seattle",
      "new_value": "Portland",
      "evidence_spans": [
        {"segment_index": 1, "turn_index": 1, "quote": "..."},
        {"segment_index": 2, "turn_index": 3, "quote": "..."}
      ]
    }
  ],
  "answer": [
    {
      "question_pair_id": "p1_state_transition",
      "evaluation_setting": "longitudinal_operation",
      "evaluation_type": "StateTransition",
      "evaluation_category": "state_reasoning",
      "question": "...",
      "expected_answer": "...",
      "gold_memory_state": "...",
      "judge_rubric": {
        "must_include": ["..."],
        "must_not_include": ["..."]
      },
      "diagnostic_checks": {
        "stale_value": "..."
      },
      "gold_provenance": [
        {"segment_index": 2, "turn_index": 3, "quote": "..."}
      ]
    }
  ]
}

The injection stage keeps the questions while replacing the short evidence history with a longer conversations list. Its injection_metadata records where evidence and distractors were inserted, their source sessions, and the seed-controlled sampling layout.

Repository Layout

MemOps/
├── 1-generate_background.py
├── 2-generate_evidence_conversation_w_verity.py
├── 3-generate_distractor_facts.py
├── 4-inject_evidence_into_irrelevant_conversation.py
├── 5-test_operation_metrics.py
├── 5.5-evaluate_operation_metrics.py
├── generated_result/
│   ├── 1-background/
│   ├── 2-evidence_conversation/
│   ├── 2-evidence_trace/
│   ├── 3-distractor_facts/
│   └── 4-inject_evidence_with_distractors/
├── info/
│   └── topics.md
├── template/
│   ├── generation prompts
│   └── verifier prompts
├── img/
│   └── generation_pipeline.png
└── requirements.txt

📝 Citation

If you use MemOps in your research, please cite our paper:

@misc{hao2026memops,
  title         = {MemOps: Benchmarking Lifecycle Memory Operations in Long-Horizon Conversations},
  author        = {Hao, Xixuan and Zhang, Zeyu and Lin, Zehao and Sun, Yihang and
                   Guo, Ziliang and Zhang, Xichong and Liang, Yuxuan and Xiong, Feiyu and
                   Li, Zhiyu},
  year          = {2026},
  eprint        = {2607.12893},
  archivePrefix = {arXiv},
  primaryClass  = {cs.AI},
  url           = {https://arxiv.org/abs/2607.12893}
}

License

MemOps is released under the MIT License.

Acknowledgements

MemOps uses UltraChat as the source of unrelated carrier conversations for longitudinal injection.