Memory-R1: Reinforcement Learning Enhanced Memory for LLM Agents

👥 Authors

Sikuan Yan¹*, Xiufeng Yang²*, Zuchao Huang¹, Ercong Nie¹, Zifeng Ding³, Zonggen Li⁴, Xiaowen Ma¹, Hinrich Schütze¹, Volker Tresp¹, Yunpu Ma¹✉️

¹Ludwig Maximilian University of Munich, ²Technical University of Munich, ³University of Cambridge, ⁴University of Hong Kong

---

🚧 Status: Code coming soon. Please ⭐ star the repo to stay updated!

---

🔥 Research Highlights

Large Language Models (LLMs) remain stateless, constrained by limited context windows that hinder long-horizon reasoning. Naive memory systems rely on heuristics for storing and retrieving information, which often leads to fragmentation or noise.

Figure 1: Motivation. Vanilla memory managers misinterpret multi-session facts (left/middle), issuing DELETE+ADD. Memory-R1 instead consolidates with UPDATE (right), and the Answer Agent distills only relevant memories for reasoning.

Memory-R1 addresses these challenges with two RL-trained agents:

• Two-agent architecture: an RL-trained Memory Manager (ADD / UPDATE / DELETE / NOOP) and an Answer Agent (Memory Distillation + Answer).
• Data efficiency: achieves stable improvements with as few as 152 QA pairs.
• Cross-backbone generalization: works consistently across LLaMA-3.1-8B and Qwen-2.5-7B.
• State-of-the-art results on LoCoMo: relative improvements of +48% F1, +69% BLEU-1, and +37% LLM-as-a-Judge over the prior best.

Figure 2: Memory-R1 pipeline. Stage 1 (blue): Memory Manager constructs and updates the memory bank. Stage 2 (green): Answer Agent performs memory distillation and reasoning.

---

📊 Results on LoCoMo

Model / Backbone	F1 ↑	BLEU-1 ↑	LLM-as-a-Judge ↑
Mem0 (baseline)	30.4	22.2	45.7
Memory-R1 (PPO)	41.1	32.9	57.5
Memory-R1 (GRPO)	45.0	37.5	62.7

📖 Paper

If you use Memory-R1 in your research, please cite:

@article{yan2025memory,
  title={Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learning},
  author={Yan, Sikuan and Yang, Xiufeng and Huang, Zuchao and Nie, Ercong and Ding, Zifeng and Li, Zonggen and Ma, Xiaowen and Sch{\"u}tze, Hinrich and Tresp, Volker and Ma, Yunpu},
  journal={arXiv preprint arXiv:2508.19828},
  year={2025}
}