Backboard LongMemEval Benchmark

February 10, 2026 · View on GitHub

Backboard

We ran Backboard against the full LongMemEval benchmark — 500 questions testing long-term conversational memory across 6 categories.

Overall accuracy: 93.4% (467/500 correct)

Results

Category	Questions	Correct	Accuracy
Single-Session (Assistant)	56	55	98.2%
Single-Session (User)	70	68	97.1%
Knowledge Update	78	73	93.6%
Multi-Session	133	122	91.7%
Temporal Reasoning	133	122	91.7%
Single-Session (Preference)	30	27	90.0%
Total	500	467	93.4%

Cross-Evaluator Accuracy

We evaluated the same answers with 4 different LLM judges to make sure results are robust and not dependent on a single evaluator.

Category	GPT-4	GPT-4o-mini	Gemini 3 Pro	GPT-5.2
Single-Session (Assistant)	96.4%	98.2%	96.4%	96.4%
Single-Session (User)	95.7%	97.1%	95.7%	95.7%
Knowledge Update	89.7%	93.6%	93.6%	93.6%
Multi-Session	94.0%	91.7%	92.5%	89.5%
Temporal Reasoning	93.2%	91.7%	92.5%	92.5%
Single-Session (Preference)	80.0%	90.0%	80.0%	66.7%
Overall	92.8%	93.4%	92.8%	91.2%

The main results table uses GPT-4o-mini as the evaluator (highest overall accuracy). Bold values in the cross-evaluator table show the best score per category.

How It Works

The benchmark is pretty simple:

Feed conversations — Send past chat sessions to Backboard (with memory=Auto so it extracts and stores memories)
Ask a question — Let Backboard retrieve relevant memories and answer (with memory=Readonly)
Score the answer — Compare against ground truth using LongMemEval's evaluation protocol

Each question gets its own isolated assistant and thread, so there's no cross-contamination between tests.

Architecture

LongMemEval Dataset (500 questions + conversation histories)
                    │
                    ▼
        ┌───────────────────────┐
        │   Backboard Runner    │
        │   (Autocomplete API)  │
        └───────────────────────┘
                    │
        ┌───────────┴───────────┐
        ▼                       ▼
  Feed Sessions           Ask Question
  (memory=Auto)          (memory=Readonly)
        │                       │
        └───────────┬───────────┘
                    ▼
            Predictions
                    │
                    ▼
        ┌───────────────────────┐
        │  LLM Judge Evaluation │
        │  (4 models, majority) │
        └───────────────────────┘
                    │
                    ▼
          Accuracy by Category

Dataset

We used the s_cleaned dataset, which has ~40-50 conversation sessions per question (roughly ~115k tokens each). The benchmark model was gpt-4.1 via OpenAI.

Question Type	What It Tests
Temporal Reasoning (133)	"When did X happen?" / ordering of events
Multi-Session (133)	Info scattered across many conversations
Knowledge Update (78)	Facts that changed over time
Single-Session User (70)	User-stated facts from one conversation
Single-Session Assistant (56)	Assistant-stated facts from one conversation
Single-Session Preference (30)	User preferences mentioned in passing

How we ran this benchmark: For this evaluation, we fed the maximum possible conversation history in a single message — that's how competitors in this space (Supermemory, Mastra, etc.) are running their benchmarks too, so we kept it apples-to-apples.

What real-world usage looks like: In practice, each message in a LongMemEval haystack session would be its own separate Backboard message — the way actual users interact in a real app. We're currently running the benchmark this way and will share those results as soon as they're ready. We're optimistic about it!

Running It Yourself

Setup

git clone <repo-url>
cd <repo>
python -m venv .venv
source .venv/bin/activate
pip install -e .

# Get the LongMemEval evaluation code
git clone https://github.com/xiaowu0162/LongMemEval.git LongMemEval
pip install -r LongMemEval/requirements-lite.txt

# Download datasets
huggingface-cli download xiaowu0162/longmemeval-cleaned \
    --local-dir data --include "*.json"

# Configure API keys
cp .env.sample .env
# Add your BACKBOARD_API_KEY and OPENAI_API_KEY

Run the Benchmark

# Quick test with a single question
python scripts/run_backboard_benchmark.py --id e47becba --evaluate

# Run a subset
python scripts/run_backboard_benchmark.py --limit 50 --evaluate

# Full benchmark (all 500 questions)
python scripts/run_backboard_benchmark.py --evaluate

Evaluate with Different LLM Judges

# Direct OpenAI evaluation
python scripts/evaluate_run.py --run-id knowledge-update --eval-model gpt-4

# Via Backboard (any provider)
python scripts/evaluate_run.py --run-id knowledge-update \
    --eval-provider openrouter --eval-model google/gemini-3-pro-preview

Generate Reports

# Multi-evaluator comparison report
python scripts/generate_report.py --multi-eval --output reports/comparison.html

Project Structure

├── src/                    # Core source code
│   ├── backboard_client.py     # Backboard API client
│   ├── backboard_runner.py     # Benchmark runner
│   ├── dataset_loader.py       # Dataset loading
│   └── config.py               # Configuration
├── scripts/                # CLI scripts
│   ├── run_backboard_benchmark.py
│   ├── evaluate_run.py
│   └── generate_report.py
├── data/                   # LongMemEval datasets (gitignored)
├── runs/                   # Benchmark results
│   ├── consolidated_results.json
│   ├── knowledge-update/
│   ├── multi-session/
│   ├── single-session-assistant/
│   ├── single-session-preference/
│   ├── single-session-user/
│   └── temporal-reasoning-report/
└── reports/                # Generated HTML reports

Requirements

Python 3.10+
Backboard API key
OpenAI API key (for LLM judge evaluation)