OAKS: Online Adaptation to Continual Knowledge Streams

This repository contains the datasets for the paper "Can Large Language Models Keep Up? Benchmarking Online Adaptation to Continual Knowledge Streams" (arxiv), accepted to ACL 2026 Main.

• OAKS-BABI (OAKS-B): A synthetic dataset derived from the BABILong benchmark. Questions focus on tracking, counting, bridge, and comparison across evolving facts. Contains 1.2k questions.
• OAKS-Novel (OAKS-N): A human-curated dataset sourced from 19 public domain novels with rich narratives and dynamically interacting characters. Contains 870 multiple-choice questions (avg. 5.5 options).

Code to run the OAKS evaluation and a detailed explanation of the OAKS datasets can be found on our project page.