What does a memory system know about itself?

Benchmark

LongMemEval-S

Our retrieval

R@5 0.927 (/search), n=150 stratified

Reader ceiling

0.868 true-oracle (gold present) — Opus|preference + canonical judge, n=500 — essentially the field's GPT-4o oracle (0.870). The earlier 0.61 was the retrieval-limited substrate.

Judge

gpt-5.3-chat + canonical type-specific prompts (NOT gpt-4o-2024-08-06)

Posture

diagnostic decomposition, not a leaderboard claim

What this table is: the field column is self-reported leaderboard numbers — mixed metrics (R@5 vs oracle QA vs E2E QA, which are three different tests), mixed answer models (answer-model choice alone swings ~24pp), and mixed verification. It is context, cited with caveats — not a controlled comparison. For that, read the apples-to-apples table above (our own harness, identical conditions). The Verif. column below tells you who measured each number:

† engram-2 = github.com/199-biotechnologies/engram-2 (Paperfot AI) — a Rust CLI: SQLite + FTS5/BM25 + Gemini Embedding 2 + RRF + Cohere rerank, self-reporting 99.0% R@5 on LongMemEval-S. It returns verbatim source chunks (with an optional LLM claim/entity extraction pass). ai-memory = github.com/alphaonedev/ai-memory-mcp (FTS5 + embeddings, 97.8% R@5). The “Engram” name is overloaded: the same org also ships 199-biotechnologies/engram (MCP server, BM25+ColBERT+KG, 98.1% R@5), and the ENGRAM (paper) row is a separate, extraction-based academic system (arXiv:2511.12960, 71.4% QA). Storage paradigms marked unstated are ones we could not pin to a primary source — left blank rather than guessed.

‡ OMEGA R@5 0.900 is our own measurement — the first independent head-to-head: OMEGA run on the identical n=150 stratified LongMemEval-S subset, session-level R@K, same retrieval definition as mempalace (which scores 0.927 on that subset — OMEGA trails by 2.7pp / 4 questions). It is not comparable to OMEGA’s self-reported 95.4%, which is a different metric (E2E QA, not R@5) with a stronger answer model (GPT-4.1; answer-model choice alone swings ~24pp). The on-harness R@5 is the apples-to-apples number. (Note: OMEGA ships without its bge-small embedding model and silently falls back to keyword-only FTS5; we ran omega setup --download-model to restore semantic retrieval before measuring, so this is the fair number, not a crippled one.)

§ mempalace daemon on LoCoMo — age-fused QA 0.388 (n=250 stratified, isolated scratch palace, never prod), drawer-level R@5 0.556. An A/B isolating the KG: restoring the graph-hydration fix (palace-daemon#202) moved QA +1.2pp over the vector-only fallback (0.376) with identical drawer-R@5 — so on LoCoMo the age-fused graph half adds ~nothing to top-5 retrieval (the graph-only hits don’t displace the vector top-5). The flat adapter scores the same QA (0.384), confirming the substrate, not the path, sets it. LoCoMo QA here (~0.39) is an our-harness figure (gpt-5.3-chat reader) — not comparable to the field’s 0.75–0.93 (stronger answer models + looser graders; cf. the event-ordering Kendall-τ vs binary-judge mismatch).

‖ The deployed-E2E column is a retrieval-breadth ladder (real /search→reader→judge, opus reader, n=150): limit=5 0.567 (85/150) → limit=20 0.740 (111/150) → limit=50 0.760 (114/150). Deployed QA was retrieval-limited at limit=5; widening to 20 recovers +17.3pp and it plateaus by 50 — concrete proof the gap to the 0.868 oracle ceiling is retrieval breadth, not the reader. (One content-filtered judge call at limit=50, qid 95228167, is counted wrong — a conservative floor; scored correct it is 0.767, which doesn’t touch the plateau.) These are a live re-query and supersede the 05-29 cached 0.610 (pre-retrieval-drift pinned context). Only the 0.868 ceiling sits under oracle QA (gold handed to the reader, retrieval bypassed) — the like-for-like to the field’s GPT-4o 0.870 oracle (techempower-org/multipass-structural-memory-eval#117).

Read the Storage column first — it is the axis a LongMemEval leaderboard hides. mempalace is verbatim-first: it stores the raw turns and never summarizes (“store everything, then make it findable”), solving retrieval separately. It is not alone in that — True Memory (a paper-only design, arXiv:2605.04897, no code release) is also verbatim-first and reports 87.8% QA, so the paradigm is not what caps the leaderboard. The rest of the field sits on the other side of the split: hybrid systems that selectively extract while keeping some raw history (Mem0’s self-editing vector+graph+KV, Zep/Graphiti’s bi-temporal knowledge graph, Letta’s tiered memory), and pure extraction (Hindsight stores structured, time-aware facts instead of raw logs). So no, these are not all verbatim-first — but mempalace’s verbatim peers are real: True Memory, traditional RAG, and the R@K-only ChromaDB baselines (agentmemory, engram-2, ai-memory at 95–99% R@5). The paradigm split is taken from the MemPalace landscape survey, not assigned by us.

The honest read: on retrieval mempalace is competitive — R@5 0.927 sits mid-pack against the field's R@K leaders (96–99% for ChromaDB-baseline systems; mcp-memory-service 80–86%). On QA we deliberately publish no full-pipeline leaderboard number (that needs a field-standard answer model + the canonical gpt-4o-2024-08-06 judge we don't have). The apples-to-apples axis is the oracle — and an earlier reading of this card got it wrong, which is worth stating plainly: we reported 0.61 and called the 26-point gap to GPT-4o's 0.87 oracle a reader limit. It isn't. That 0.61 was retrieval-limited — the context reaching the reader came through /search at limit=5, and for single-session-assistant questions our upstream-parity ingest (user-turns-only, which upstream itself recommends against) dropped the assistant-authored gold entirely. Hand the reader the gold instead (true oracle, evidence sessions verbatim) and five of six categories recover, not just assistant: single-session-assistant 0.32→0.98 (ingest), temporal 0.36→0.75, knowledge-update 0.70→0.91, multi-session 0.71→0.87, single-session-preference 0.80→0.93 (all retrieval); only single-session-user carried over. That lifts the overall from 0.610 to a 0.868 reader ceiling — within 0.2pp of the field's 0.870. So the old 26-point “reader” gap decomposes as ~7pp ingest artifact (the dropped assistant turns; fixed → 0.684) + ~18pp retrieval breadth (limit=5 + chunking never delivering the gold) + ~0pp genuine reader. The reader was never the bottleneck — once the gold is in context it essentially matches the field's oracle; getting the gold to it is the whole game. The route here is itself a finding: an earlier reader floor-lift — three category-specific prompt clauses on the pinned /search context — returned a clean null (net −2 to +3 questions), which is precisely what flagged that the binding constraint was substrate, not prompt; the true-oracle test then confirmed it. That decomposition is the SME finding the leaderboards can't show. And the structural categories — ingestion integrity (Cat 4), gap-detection (Cat 5), ontology coherence (Cat 8), invocation discipline (Cat 9) — have no competitor analogue at all.

Taken next to the field’s LoCoMo QA (0.75–0.93), our ~0.38 looks alarming. It isn’t a verbatim-substrate failure — it is four measurement choices stacked on top of each other, none of which the leaderboard numbers share. Reading them in order:

1 — the substrate isn’t the limit. The flat adapter (plain verbatim + vector retrieval) scores 0.384; the full age-fused daemon (pgvector + AGE knowledge graph) scores 0.388 on the same n=250 stratified subset. If the verbatim store were the weakness, the graph-augmented path would pull ahead — it doesn’t. Whatever caps LoCoMo here sits above the substrate, in the retrieval-and-judge layer that every system shares. 2 — it’s a retrieval-recall ceiling. Drawer-level R@5 on LoCoMo is ~0.44: more than half the time the gold turn isn’t in the top-5 the reader sees, so the QA number is bounded by recall, not by the reader’s ability to answer. 3 — our judge is strict. It is a binary, abstention-aware grader with no partial credit and no looser string-overlap leniency — the same discipline we hold mempalace to everywhere else. Field LoCoMo numbers frequently use softer graders (and much stronger answer models), and answer-model choice alone swings results by ~24pp. 4 — we include the adversarial split. Our subset keeps the adversarial LoCoMo questions that many field reports quietly drop; those are exactly the ones designed to defeat retrieval.

Benchmark

BEAM (bucketed by haystack size)

Substrate

flat-local mempalace

Reader

gpt-5.3-chat

Judge

o4-mini

BEAM is the one benchmark here with a genuine regime shift — a cliff, then a plateau. At 100K the whole haystack fits in context, so the reader operates in a full-context regime and the needle abilities are intact. The cliff is 100K→500K (0.649→0.487, −0.162): the system crosses into a real retrieval regime and the needle-dependent abilities collapse — info-extraction 0.85→0.40, temporal 0.75→0.26, multi-session 0.60→0.24 — while the retrieval-light ability holds flat (summarization 0.87 in both, because summarizing doesn’t need a specific needle). Then it plateaus: 500K→1M is essentially flat (0.487→0.471, −0.016) and a couple of needle abilities even tick up. So once you’re retrieval-limited at a low top-K, haystack size barely matters — the bottleneck is retrieval breadth, not corpus scale. That asymmetry is the tell: a retrieval-recall ceiling, not a reader ceiling — the same finding as our LongMemEval decomposition (the reader is fine once the gold is in context; getting it there is the whole game).

Corpus

jp-realm-v0.1 (n=30)

Backend

postgres + pgvector + AGE

Pattern

RLM orchestrator A/B

Models

qwen3.5:4b vs gemma4:e4b

The published Tau2 tool-use benchmark gap between qwen3.5:4b and gemma4:e4b is +37.7 points in qwen's favour. On our independent 30-question Cat 9a-shaped RLM experiment, the empirical recall gap landed at +30.0pp at n=5 and +33.3pp at n=20 — Tau2 predicts the gap to within ~5pp. Tau2 is therefore a useful prior when picking an orchestrator model for any RLM-as-tool-use experiment, and a much stronger predictor than parameter count.

Corpus

longmemeval_oracle.json

n

500 questions

Adapter

mempalace-daemon (postgres + pgvector + AGE)

Endpoint

POST /search (default)

Reader

o4-mini

Judge

gpt-5.3-chat

Content rules

upstream-exact

Ingest

per-question wing isolation

The headline flips once the R@5 matcher is correct: the daemon finds the gold session in the top 5 97% of the time. The 38-point R@5→QA gap looked like a reader failure — but the true-oracle correction (see the comparison card) showed it is substrate: “R@5 found the session” doesn't mean the gold content reached the reader (limit=5 + chunking left it out), and with the gold actually present QA rises to a 0.868 ceiling. cat_6 (temporal) is the only category below the R@5 ceiling (90.55%) and also the worst QA (40.94%): its evidence is the most fragmented across the limited context. cat_1_negative's QA (90%) clears its R@5 because a correct abstention is the right answer regardless of what was retrieved.

Corpus

longmemeval_oracle.json

n

500 questions

Adapter

mempalace-daemon

Endpoint

POST /search/age-fused

Reader

o4-mini

Judge

gpt-5.3-chat

The corrected matcher (techempower-org/multipass-structural-memory-eval#98) turns this card from a mystery into a clean finding. /search/age-fused retrieves the gold session at R@5 = 90.2% — almost as well as the default endpoint. Yet QA accuracy is 17.4%, a 73-point R@5→QA gap. The cause is the snippet width: age-fused returns ~457 chars of context per query (5.5× less than default's 2539). The gold session is in the results, but the reader is handed too thin a slice of it to answer. cat_2c (multi-session) collapses to 0% QA at 98% R@5 — the starkest illustration: every gold session retrieved, none answerable from the snippet. cat_1_negative rises to 100% because thin context makes the reader abstain, which is correct for unanswerable questions.

Corpus

longmemeval_oracle.json

n

500 questions

Adapter

familiar (Hybrid v4 + rerank, fork)

Reader

o4-mini

Judge

gpt-5.3-chat

Familiar.realm.watch is a separate memory system in this household — same corpus, different retrieval and reader architecture. The second-adapter discipline is the point: a single reading is a single corpus, and brittle defaults hide on any single corpus. With the corrected matcher (techempower-org/multipass-structural-memory-eval#98) the finding inverts from an earlier reading: Familiar's R@5 (28.6%) is the lowest of the three legs, not the highest. Its retrieval is genuinely weaker than daemon-direct's 97%. And unlike the daemon legs, Familiar's QA (31.0%) slightly exceeds its R@5 — the reader does fine with what it gets; the limit is what reaches it. A different shape from #44/#45, where retrieval finds the session but — as the true-oracle correction later showed — the gold content still often didn't reach the reader. Both point the same way: the limit is what reaches the reader, not its reasoning.

In-domain

LongMemEval-S via MemPalace's own bench (n=500)

Cross-domain

jp-realm-v0.1 (n=30, covered n=29)

Encoders

all-MiniLM-L6-v2 base vs adaptmem FT-300

Stacks

raw · +FT-300 · +hybrid_v4+FT-300

Domain-adaptive encoder fine-tuning is a real retrieval lift in-domain. On LongMemEval-S through MemPalace's own bench, adaptmem's FT-300 moves R@5 0.966→0.980 and R@1 0.806→0.862; stacked with hybrid_v4 retrieval the gains compose to R@1 0.916 / R@5 0.990 — encoder fine-tune (layer 2) and hybrid retrieval (layer 3) operate on different failure modes and add independent lift. Our own katana FT-300 reproduction hit R@5 = 1.000 on the held-out 200. So the encoder is not useless — quite the opposite.

What our jp-realm reading shows is narrower: the lift does not transfer cross-domain. A FT-300 encoder trained on conversational memory, dropped onto JP's personal KB, gives best delta +1.73pp at R@1 (R@5 flat) against a predicted +30–33pp — a clean cross-domain null, exactly the open question the orthogonal-layers note flagged. And it doesn't change the end-to-end picture either: on oracle LongMemEval retrieval is already ~0.974 while the reader leaves a ~45pp gap (techempower-org/multipass-structural-memory-eval#116). Encoder tuning is a legitimate retrieval improvement, but it is not the lever for end-to-end QA — what reaches the reader is.

Corpus

longmemeval_s_cleaned.json

n

150 (stratified, 25 per question_type)

Adapter

mempalace-daemon

Endpoints

/search vs /search/age-fused

Scoring

drawer-id R@K (techempower-org/multipass-structural-memory-eval#98), retrieval-only

On a representative, category-stratified sample, age-fusion shows NO significant R@5 gain over plain /search (Δ = −1 question of 150; R@1 +2 questions). The +2.0pp R@5 “win” seen on an earlier n=100 first-100 slice did not replicate — that slice was single-session-dominated (the S corpus is question_type-sorted; fixed via --stratify-by in techempower-org/multipass-structural-memory-eval#122). Per-category (n=25 each, ±1-question noise → directional hypothesis only): age-fusion helps temporal-reasoning + knowledge-update, hurts single/multi-session recall.

Corpus

longmemeval_oracle.json (pinned context, retrieval held fixed)

n

500

Readers

o4-mini, gpt-5.3-chat

Judge

gpt-5.3-chat

Retrieval ceiling

R@5 97.4%

With the search-default context pinned (R@5 0.974), both readers answer correctly only ~50% — a ~45pp R@5→QA gap. Correction (see the comparison card): this gap is substrate, not the reader. The true-oracle test later showed that “R@5 found the session” does not mean the gold content reached the reader — limit=5 + chunking (and, for assistant-authored answers, the user-only ingest) often left the answer out of the context entirely. Hand the reader the gold verbatim and QA rises to a 0.868 ceiling, near GPT-4o's oracle. So the ~50% reflects what reached the reader, not a reasoning limit. Age-fused context is harder still (QA 43–47%). Self-judge caveat: gpt-5.3-chat judges its own family on its own reader leg, which may inflate it.

Corpus

pinned oracle (stratified 150, search-default)

Readers

claude-opus-4-8 (Bedrock), o4-mini, gpt-5.3-chat

Prompt

baseline

Judge

gpt-5.3-chat

Opus 4.8 — the strongest reader — scored the worst overall (39.3%). But it is not a capability gap: Opus is the best reader on well-posed direct recall (single-session-user 84%, the highest of any reader on any category) and collapses only on mis-specified categories (preference 0%, assistant 12%). It is penalized for following the baseline prompt literally (over-abstaining where the gold answer is an inferred preference) and for thoroughness (reporting both old+new values → judged PARTIAL; answers 2× longer). The 45pp gap is prompt + judge design, not reader capability. Both follow-ups are now published below: the prompt-axis fix and the Opus-as-judge re-scoring.

Corpus

pinned oracle (stratified 150, 25/type, search-default)

Readers

claude-opus-4-8 (Bedrock), o4-mini

Prompt axis

baseline → committed → preference

Judge

gpt-5.3-chat (not yet canonical — floor only)

Context

full oracle

“Opus is the worst reader” was a prompt artifact. With the preference-tuned prompt, Opus goes from worst (0.360) to best of any config (0.593) — a +23pp swing from changing nothing but the prompt. The entire gap lived in one category: single-session-preference, where Opus scored 0.04 under the baseline prompt (it refused to make a recommendation — the “say I don’t know” instruction over-fired on inference questions) and 0.76 under the preference prompt (+72pp). That single category drags the overall from 0.36 to 0.59. This is the reader-harness fix; the judge is still gpt-5.3-chat (not the LongMemEval-canonical type-specific judge), so 0.593 is a floor — the canonical-judge work compounds on top. Bottom line: the #116 reader gap is prompt design, not model capability.

Repo

palace-daemon (#46)

Corpus

live palace, ~375k drawers

Backend

postgres + pgvector (familiar)

Model

ms-marco-TinyBERT-L-2-v2 (nano, CPU)

Query set

12 known-item, 11 usable, pool=20

Pattern

same pool, ordering-only A/B

A clean ordering-only A/B — retrieval is held constant, only the final sort changes (distance vs FlashRank score) on the same candidate pool. MRR lifts +15.3% (run #1) and +23.1% (run #2), driven by a buried answer rescued from rank 7 to rank 1 plus two smaller promotions; 7 of 11 queries were already optimal and rerank correctly left them alone. R@5 is a wash and R@10 is untouched, so MRR is the load-bearing metric for this known-item set. Rerank latency stayed at 47 ms mean (126 ms under host load), worst single request 557 ms — well inside a 1 s budget on CPU. Verdict: keep nano, A/B a larger model next.

Corpus

live familiar palace (postgres + AGE)

Query set

12 labeled known-item (palace-daemon PR #64)

Method

in-process searcher, FlashRank OFF, weights set per sweep point

Knob

PALACE_HYBRID_VECTOR_WEIGHT / _BM25_WEIGHT (mempalace#342)

The #111 acceptance criterion — a documented hybrid weight achieving R@5 ≈ 1.000 without regressing MRR vs union/vector — is met by vector_weight 0.85 / bm25_weight 0.15. The default 0.6/0.4 hybrid had regressed MRR to 0.785 (below the 0.808 union/vector floor); 0.85/0.15 lifts it to 0.833 — +2.5pp over the floor, +4.8pp over the default-hybrid regression — while R@5 stays pinned at 1.000. MRR turns out to be non-monotonic in vector_weight: the earlier “graph promoted one query and demoted another” reading was wrong; the real mechanism is the convex blend over-weighting BM25 at 0.4. The weight ships as an env knob (mempalace#342) — the deployed default stays 0.6/0.4 because n=12 is below the n≥25 bar to flip a production default.

Probe set

200 git-derived probes (same set as the #162 fusion A/B)

Substrate

scratch daemon seeded with the mempalace git/docs corpus

Method

daemon /search/hybrid per-request rerank flag, read-only A/B

This is the SME-side rerank A/B — distinct from the palace-daemon FlashRank known-item A/B above (a 12-query run that found a +15–23% MRR lift on a different, answer-buried query shape). On a corpus-seeded scratch daemon — the git/docs targets re-ingested so the relevant set is actually present — the verdict is the opposite: cross-encoder rerank is neutral-to-slightly-negative. R@10 is identical at 0.60 across all three legs (rerank only reorders the top-K; it cannot recall what vector retrieval missed), MRR is slightly hurt by rerank (0.299→0.293), and the bigger MiniLM-L-12-v2 is both worse on MRR and 3× the latency (1523 ms vs 475 ms p50). Recommendation: keep rerank OFF / opt-in, do not promote the bigger model. This is the 4th independent reading that the structural/rerank layer doesn’t lift retrieval — the vector backbone is the lever (with the age-fusion null ×3: Cat 2c/7 graph-RRF tax, #91 age-fusion R@5, and the #111 graph-leg-inert finding).

Repo

palace-daemon (#47)

Corpus

live palace, ~375k drawers

Backend

postgres (familiar:8085)

Method

gzip-NCD, block rolling window=20

Sample

35 (wing,room) groups, n=280

Scorer

production novelty.score_novelty

The gzip-NCD novelty_score on the live corpus is continuous and right-skewed — not bimodal: a broad novel hump across 0.55–0.85, a long redundant tail below ~0.20, and the middle band populated throughout with no empty valley. Per-room baselines diverge by ~0.37 in median (discoveries 0.30 vs planning 0.68), so a single global NCD cut mislabels content; the calibration recommends per-room percentile thresholds — redundant at p15(room), novel at p60(room).

Corpus

pinned oracle (stratified 150, search-default)

Readers

claude-opus-4-8 / o4-mini / gpt-5.3-chat (same answers as Pass B)

Judge re-scored

gpt-5.3-chat → claude-opus-4-8

Method

same hypotheses, different judge

To isolate the judge variable, the identical Pass B reader answers were re-graded with Opus-4.8 as judge (the original judge was gpt-5.3-chat). Every reader lifts modestly — +1–3pp — mostly by rescuing single-session-preference questions (the Opus reader's preference category moves 0.00→0.12). But the Opus judge does not change the ranking and does not rescue the Opus reader, which stays the lowest at 0.420. So the judge confound is real but small: the dominant factor in the reader gap is prompt design, not judge strictness. The prompt-fix sweep is published — see the prompt-axis card above.

Corpus

LongMemEval-S oracle, n=500 (all 6 types + 30 abstention)

Reader

gpt-5.3-chat (held constant, baseline prompt, full context)

Judge

gpt-5.3-chat + canonical type-specific prompts (NOT gpt-4o-2024-08-06 — not deployed here)

Method

same reader answers, paraphrased rubric → verbatim LongMemEval templates

The companion card swaps the judge model (Opus-as-judge) and finds it doesn't help. This card swaps the judge prompts: porting LongMemEval's verbatim type-specific templates — in particular the rubric-based preference template the old paraphrased judge lacked. That un-collapses single-session-preference (+20.0pp / +26.7pp, the largest single-category move) and removes the spurious-ABSTAIN noise (the old judge emitted 34 ABSTAIN labels, 21 on non-abstention temporal questions; the canonical binary judge can't). The stricter, more faithful templates pull KU and multi-session down a few points — that is the old judge having over-credited via a softer rubric, not a regression. The headline: the overall did NOT move toward the published 87% oracle — corrected, abstention-credited it is 0.562 / 0.510. The judge-prompt confound was real (preference collapse and label noise were genuine measurement artifacts) but it is not the bulk of the 35pp gap. The residual ~31–36pp is reader/substrate, not the scorer — and the true-oracle test (see the comparison card) later resolved which: it is substrate (what reaches the reader), not the reader's reasoning.

Disclosure. Judge is gpt-5.3-chat with canonical prompts, not the canonical gpt-4o-2024-08-06 snapshot (not deployed on this resource) — the fix is the prompts, not the model. 2 ERROR rows per run from an Azure content-filter trip (graceful retry-then-ERROR by design; counted as wrong, a 0.4% asymmetry vs the confounded denominator). Absolute numbers are understated by the aggregator’s ABSTAIN dead-code bug (#148) — it never credits a correct ABSTAIN — but that depresses both runs identically, so the deltas are valid.