A benchmark gate that catches small regressions

A benchmark you run once is a screenshot. A benchmark that fails a pull request is a guard. This post describes the regression gate in Aletheia: what it measures, how its baseline was set, and what it cannot tell you.

The gate

bun bench/gate.mjs builds a seeded synthetic corpus of 50,000 rows across ten topics, runs 100 queries five times, and compares two numbers with a stored baseline:

• Recall@5: whether the right memory is in the top five results. Baseline 0.90.
• p95 latency: the time within which 95 percent of queries answer. Baseline about 7.8 ms (7.759 ms), recorded on 2026-10-28 on linux x64 bun 1.3.10. Each baseline figure is the median of five measurements.

The check fails if Recall@5 falls by more than 0.01, or if p95 latency rises by more than 20 percent. The latency check is enforced only when the baseline p95 is at least 5 ms.

Why seeded

Random data gives a different answer each run, so a drop could be noise. A fixed seed makes the corpus identical every time. Any change in the numbers comes from the code.

Calibrating on the real runner

A baseline set on one machine can fail on another. Two runs on a GitHub runner gave p95 latencies of 7.268 ms and 7.294 ms against the 7.759 ms baseline, about 6 percent below it. The 20 percent margin was chosen, and two runs are too few to prove it is the right width. The baseline file records the machine only as linux x64 bun 1.3.10.

What the gate cannot tell you

• It measures retrieval on synthetic data. It does not measure whether an answer was correct.
• It runs on Linux only in CI.
• It has no trend check: each run is compared with the fixed baseline.
• A synthetic corpus can hide problems that real, messy text would show.

What I would do differently

I would add a trend check. Against a fixed baseline, a slowdown that creeps up in small steps passes every run until the total passes 20 percent.

This post was drafted with AI assistance. I reviewed every claim and ran the commands shown.

About the project: Aletheia.

Mohamed Ashraf Elsaed
Technical Lead at Convertedin, Cairo. I build tools for AI coding agents and write about Laravel, Node and engineering.