Model A scores 73.5. Model B scores 70.0. It looks like A is better by 3.5 points. With 200 tasks, you can’t actually tell, and most published comparisons never say so.
Why a score can mislead
Every score comes from a limited set of tasks. Run the same two models on a different 200 tasks and both numbers would move a little. That wobble is the margin of error. If the two models’ ranges overlap enough, a 3.5-point gap could be nothing more than which tasks happened to be picked.
Small scoring choices move the number too. Say 4 of Model B’s 200 runs crashed and have no score. Count them as failures and B scores 70.0. Leave them out and B scores 71.4. Both are defensible, but they are different results, and a reader should know which one they are looking at.
16%
of 445 benchmarks reviewed by 29 experts reported any uncertainty estimate or statistical test.
Source · Measuring what Matters, 2025What Bohrin does
Step 1. Point it at your results. One line per task per model, in a .jsonl file. Use null for a run that errored. Other fields are ignored, so an existing results file usually works as it is.
1{"task_id": "t1", "model": "model-a", "score": 1}2{"task_id": "t1", "model": "model-b", "score": 0}3{"task_id": "t2", "model": "model-a", "score": 0}4{"task_id": "t2", "model": "model-b", "score": null}
Step 2. Run it. Here is the real output for two models on 200 tasks:
What that output says, in plain words
- —Each score has a range. Model A is 73.5, but anywhere from 67.0 to 79.1 is likely.
- —The two models can’t be told apart. The gap is +3.5 points, but its range runs from −4.5 to +11.5 and includes zero. “A is better” isn’t supported by this evaluation.
- —This evaluation can only see big differences. The smallest gap it can reliably detect is 11.5 points. To see a 2-point difference, you need far more tasks.
- —There is a scoring choice to disclose. Four of B’s runs errored. Counted as failures B is 70.0; dropped, it would read 71.4.
If you know the smallest difference that matters to you, say so with --min-difference 0.02 (2 points), and Bohrin tells you whether the evaluation is big enough to see it.
When tasks come from the same source, such as the same repository, they aren’t independent. Add a cluster field to each line and the ranges widen to match.
Other scoring problems it catches
- —Scores above full marks
- —Tasks one model skipped and the other didn’t
- —The same run counted twice
- —Partial credit mixed with pass/fail
- —Errored runs, and how counting them changes the result
It never skips a bad line silently. If a line can’t be read, the message names the line.
What it doesn’t do
- —It reads your results file. It doesn’t look inside the tasks, so it can’t tell you whether they measure the right thing.
- —It doesn’t run your models. It needs no model, no API key and no internet, and runs in seconds on a laptop.