Research · July 5, 2026 · 6 min read

AI agent memory benchmarks in 2026: the same-judge rule

Agent-memory leaderboards look precise — 94.8, 63.8, 91.6 — and are quietly among the least comparable numbers in the field. Before you pick a memory layer on a benchmark, learn the one rule that separates a measurement from a marketing artifact.

The two-number problem

Here's a real example of the confusion. On the LongMemEval benchmark, one vendor reports a score around 94.8 for its own system. An independent evaluation of the same system, on the same benchmark, measured it near 49.0 — and measured a competitor at 63.8, flipping the ranking entirely.

Nobody is lying. Both numbers were produced by running "LongMemEval." The scores disagree by 40+ points because a benchmark name is not a benchmark run. Everything underneath — the judge model, the grading prompt, the retrieval budget, the ingestion settings — changes the number, and vendors and independents make different choices.

Why the judge decides the score

Modern memory benchmarks grade free-text answers with an LLM-as-judge: a model reads the system's answer and the reference, and decides whether it's correct. That single choice dominates the result. Swap a strict judge for a lenient one and easy points appear. Change the grading prompt and borderline answers flip. Give the system a bigger token budget at read time and recall jumps.

So a headline score with no harness attached isn't a measurement you can act on. It's a snapshot of one lab's configuration, reported by the party with the most to gain from it.

The same-judge rule

There's a simple rule that restores meaning: a benchmark comparison only counts if every system was run through the same harness — same judge model, same prompts, same retrieval budget, reproducible, with the code linked so you can run it yourself. Same judge, or it doesn't count.

Comparing two vendor-reported scores from different harnesses is comparing two rulers with different inch marks. The numbers are real; the comparison is fiction.

This is why the honest players in adjacent spaces publish open harnesses and invite you to reproduce. It isn't just ethics — it's the only way a number survives contact with a skeptical engineer.

What these benchmarks even measure

Even run cleanly, the popular suites — LoCoMo, LongMemEval, DMR — measure one thing: how well a single assistant recalls facts from a single user's long conversation. That's a real and useful capability. But notice what it doesn't touch:

Concurrent writes from multiple agents to the same state.
Whether agent B's context is guaranteed to contain agent A's latest write (read-your-writes).
Per-agent access control and whether untrusted writes leak into prompts.
Freshness — how many milliseconds until a write is queryable.

A system can top LongMemEval and still have no answer for any of these. That's not a knock on the benchmark; it's a reminder that coordination lives on axes single-user recall benchmarks were never designed to test.

How we'll report numbers

Our stance, stated up front so you can hold us to it:

Same judge, or it doesn't count. Every comparison we publish is reproduced through one harness, with the code linked — run it on your own workload.
We measure the axes that matter for teams. Beyond borrowing the standard recall suites for parity, we're building a coordination benchmark for the things above — concurrent writes, read-your-writes visibility, access-control leakage, freshness under load.
No context-free headlines. A score without its harness is a claim, not evidence.

And the honest part: we don't have a leaderboard trophy to wave. v0.1 is the current target. What we have is a methodology — and a refusal to play the number-out-of-context game that makes memory benchmarks untrustworthy in the first place.

The one-line version

An agent-memory benchmark number means nothing without its harness. Ask for the judge, the prompts, and the reproduction link before you believe any of them — including ours.

We keep every comparison same-judge and reproducible on the Compare pages, with an honest "when to choose them" on each. More on the category in the 2026 landscape. Repo on GitHub.