Aletheia vs claude-mem: reproducible numbers
claude-mem had about 104,000 stars on 2026-10-29 and, excluding anonymous contributors, 297 contributors. Aletheia had 9 stars (GitHub API, 2026-10-29). If you came here for a winner on community, it is claude-mem, and it is not close. This post compares one narrow thing: retrieval quality on one dataset, with commands so you can repeat it.
What was measured
The test is LoCoMo, a set of long conversations with questions that have gold evidence. For each of 1,536 questions (LoCoMo categories 1 to 4; category 5 is left out) we ask what share of the gold evidence turns is in the top five results (Recall@5) and how high the first one ranks (MRR). This is retrieval. It is not whether an answer was correct. Answer accuracy is not measured for either project.
| Run | Recall@5 | MRR | Time per query |
|---|---|---|---|
| claude-mem 15.2.0, raw turns, from the Aletheia README (no raw file committed, date not recorded) | 36.0% | 0.285 | p50 1,699 ms, p95 2,824 ms (idle, 40 queries) |
| Aletheia, raw turns, keyword search (BM25, the default) | 55.3% | 0.482 | 2.00 ms |
| Aletheia, raw turns, hybrid search (needs the optional embeddings package) | 57.4% | 0.497 | 3.72 ms |
| Aletheia hybrid, with facts it extracted itself (rules) | 57.7% | 0.492 | 4.94 ms |
| Aletheia hybrid, with the dataset’s own facts (an experiment, not a feature) | 64.8% | 0.591 | 5.53 ms |
Read this before you quote it
- The time columns are not like for like. Aletheia’s figure is the mean of an in-process ranking call and leaves out the time to embed the question. claude-mem’s is a median and a 95th percentile for an HTTP request to its worker. The result files record no machine, so do not compare across machines.
- The rule-based fact extractor adds almost nothing. It moved Recall@5 from 57.3% to 57.7%. The gain from facts needs a model, and a first run with a 1B local model failed.
- One dataset. A different dataset can change the order.
- The claude-mem row is not backed by a committed raw file. It comes from the README and is produced by
bench/claude-mem-baseline.mjs, which saves each turn through claude-mem’s save endpoint. Normally claude-mem ingests through its own model-based observer, so raw saves may understate it. The baseline used 15.2.0, and a newer release (15.6.0 on 2026-10-29) was not tested. - There is a known gap in the evidence. A local log of the raw-turns run, which git ignores, shows 44.7% for the hybrid search, while the committed file shows 57.4%. The cause is unknown and the run was not repeated.
Repeat it
git clone https://github.com/mohamed-ashraf-elsaed/aletheia-mem
cd aletheia-mem
bun scripts/compare-bench.mjs # prints the Aletheia rows and checks each claim row (not the claude-mem row)
bun bench/gate.mjs # synthetic Recall@5 and p95 latency, checked against bench/baseline.jsonThe LoCoMo dataset is a separate download, with a SHA-256 to check, and the hybrid rows need the optional embeddings package. The steps are on the benchmark page.
Where claude-mem is ahead
A live viewer, many more integrations and a far larger community. If you want what most people use today, use claude-mem. Aletheia exists for people who want one store shared by several tools, rules synced to each tool’s file, and numbers they can re-run.
What I would do differently
I would have measured answer accuracy first, and I would commit the baseline’s raw output so the comparison is more repeatable.
Sources
- claude-mem on GitHub (stars, forks, licence), fetched 2026-10-29
- claude-mem documentation, fetched 2026-10-29
- Aletheia benchmark page, fetched 2026-10-30
