Leaderboard & public results policy
July 15, 2026 · View on GitHub
FluctlightDB does not publish headline benchmark numbers to a third-party agent-memory leaderboard today. This page explains why, what we do instead, and what would change that.
Agent-memory “leaderboards” (LoCoMo, LongMemEval)
| Fact | Detail |
|---|---|
| Canonical live leaderboard? | No — LoCoMo and LongMemEval have official datasets/papers, but no neutral site where vendors submit scores and get ranked like BEIR or MTEB |
| What Mem0/Zep/etc. publish | Mostly self-reported blog posts, PDF tables, or GitHub README percentages |
| What we publish | Frozen JSON in-repo + open reproduce scripts + honest REPRODUCIBILITY.md |
| Third-party submission status | Not submitted anywhere external for LoCoMo / LongMemEval |
Our public numbers live here:
| Artifact | Purpose |
|---|---|
benchmarks/results/paper-2026-07-09.json | Paper freeze bundle |
benchmarks/results/locomo-invented-stack-engine-2026-07-13.json (MiniLM), locomo-mpnet-engine-2026-07-15.json (mpnet) | LoCoMo cert — honest raw evidence recall, no expansion (96.8% @150 MiniLM / 97.0% mpnet, @5=72.6%/75.1% tight-k). The old 99.0% headline was ±3 neighbor-expansion inflation, deprecated. |
make reproduce-locomo | Anyone can verify LoCoMo cert locally |
| GitHub Releases + PyPI | Versioned engine, not a ranking |
What we are not doing yet (on purpose)
| Action | Status | Why |
|---|---|---|
| Mem0/Zep head-to-head blog post | Blocked | Credible only after first external LoCoMo reproduction (REPRODUCIBILITY.md) |
| Vendor roundup outreach (EverMind-style) | Not started | Waiting on independent repro of headline numbers |
| Claiming “#1 on LoCoMo leaderboard” | Never | That leaderboard does not exist as a neutral registry |
What we can submit (optional, different metrics)
| Venue | Metric type | Fit |
|---|---|---|
| BEIR leaderboard | IR nDCG@10 | SciFact 0.645 — standard IR, not agent E2E QA |
| arXiv / Zenodo | Paper + frozen JSON | Already done (DOI on README) |
| Hugging Face dataset card | Reproduce scripts | Optional future |
| Independent repro issue | “Verified by X” | Preferred — public credit in REPRODUCIBILITY.md |
When we will pursue external visibility
- First external LoCoMo match → link from README + optional comparison post on identical protocol
- Independent LoCoMo/LongMemEval repro → add to REPRODUCIBILITY table
- Co-maintainer or named reviewer → stronger bus-factor story for adoption posts
Summary
We are not withholding from a leaderboard — there is no neutral agent-memory leaderboard to submit to.
We publish reproducible artifacts instead of registry entries. External credibility comes from independent reproduction, not from us self-listing on a vendor comparison site.
See also: MAINTAINER.md · BENCHMARKS.md