We held the Guardrail benchmark to one standard: numbers that can’t be reproduced aren’t numbers, they’re marketing. Engram gets the same treatment. So we built an A/B benchmark and ran our memory claims through it.
The benchmark
A synthetic company corpus (“Acme Labs”, deterministic seed) at 25 and 100 memories, four conditions, one fresh session per question — no cross-question contamination:
| Condition | K=25 | K=100 |
|---|---|---|
| Native A1 — every memory injected via CLAUDE.md | 100% | 95% |
| Native A2 — CLAUDE.md index + topic files | 100% | 100% |
| Engram — MCP vault, semantic search | 100% | 100% |
| Control — no memory | 0% | 0% |
The control scoring zero on all 100 questions confirms the questions are unanswerable without memory — this measures retrieval, not guesswork.
What we learned
Adoption is automatic. In 100 of 100 Engram sessions, the model called engram_search on its own — the CLAUDE.md file said nothing but # Acme project. MCP tool presence was enough. Using memory over MCP needs zero configuration per project.
The failure mode we beat is injecting too much. A1’s two misses at K=100 were the same ambiguous question, both reps: with 100 memories injected at once, the model consistently anchored on the wrong one of two same-service incidents. Engram and A2 — which retrieve or read only the relevant slices — both answered correctly. Wholesale injection doesn’t degrade gracefully; retrieval scopes the answer space.
Cost honesty. At these scales Engram costs more per question than injection — $0.207 vs $0.151 at K=25, $0.247 vs $0.169 at K=100, and it’s slower (21s vs 6s). Below roughly 100-150 memories, injection is genuinely cheaper. We’re publishing that. But the growth shapes differ: injection sends the entire corpus into every question (+11% per 4× corpus, linear, and it fills a context window entirely around 4,000 memories), while Engram sends only the retrieved chunks (+4.5%). The measured trend lines cross at K ≈ 110-150 memories — and beyond that, injection stops being deployable at all. For a team vault, retrieval is the only shape that survives scale.
The environmental reading is the same shape: tokens processed are the standard proxy for inference energy, and re-broadcasting the whole knowledge base into every interaction, forever, is a linear cost with a ceiling. Engram embeds the corpus once, locally, and pays per query only for what that query needs.
The fine print
Synthetic corpus, one model, exact-match grading, 20 questions per rep with 2-3 reps — the 95% figure rides on exactly two misses, both the same ambiguous question. The full methodology, every number, and every limitation: MEMORY-COMPARISON.md. The harness is in the repo and reproducible in three commands. Total API spend across the benchmark’s entire history — including the invalidated runs we kept for honesty — was about $102.