More and more training runs use environments built by someone else. Each one comes with a grader you didn’t write and can’t easily inspect. If that grader has a gap, the model you train on it learns the gap, and you find out only when the model behaves badly somewhere else.
Why a bought grader is a risk
A vendor can tell you their graders are solid. What’s hard is checking that claim. A report that says “no issues found” looks the same whether everything was tested or almost nothing was.
And the cost of a gap isn’t limited to one bad score. Research has found that models which learn to cheat their graders during training can carry that behaviour into other situations.
43×
more reward-hacking attempts by frontier models when the scoring code was visible to them.
Source · METR, 2025Sabotage
Models that learned to hack real coding environments during training went on to show alignment faking and sabotage elsewhere.
Source · Anthropic, 2025What to ask a vendor for
Bohrin gives you and your vendor the same checks and the same words for the results. Ask for:
- 1.The Coverage Score, with its margin of error and number of tasks. A score from 5 tasks and a score from 500 are not the same claim.
- 2.Correct-work acceptance beside it. A grader that rejects everything has a perfect Coverage Score.
- 3.The “not run” list. Every known kind of gap that applies and wasn’t tried. Not tried is never the same as passed.
- 4.A reproduction script for every finding. So you can see each one happen yourself.
- 5.The version of Bohrin’s checks. Scores are only comparable within the same version.
Every known way a coding grader gets fooled has a permanent name, like BGW-102 for “exit before the checks”. It works the way CVE numbers work for security bugs: when a vendor’s report and your own checks say BGW-102, you are talking about the same thing.
Re-run any finding yourself
Every proven finding comes with a small Python script named after it. It is the proof, and it is written so you don’t have to trust Bohrin, or the vendor:
- —It doesn’t need Bohrin installed, and makes no network call.
- —It contains the exact submission, and checks it wasn’t altered before running.
- —It runs the grader, unchanged, on the reference answer and on the cheat, three times each, each run in its own process with a time limit.
- —It prints one line saying what happened.
1uv run reproduce_BF-GZW6R2M9CJ.py # or: python reproduce_BF-GZW6R2M9CJ.py
runs is the cheat: full marks all three times. baseline_runs is the correct answer, which also passed, so the grader works in this setup. The last word is the verdict:
- —
reproduced: the gap is real. The grader paid full marks for the cheat every time. - —
not-reproduced: the grader rejected the cheat. The finding stops counting as proven. - —
flaky: the grader’s verdict changed between runs. - —
baseline-failed: the correct answer failed its own grader, so nothing can be concluded. - —
error: the grader couldn’t be run. The output says why.
How Bohrin keeps itself honest
- —No accusation without proof. A submission is only called wrong when that is certain without asking the grader. Anything less certain is shown as a lead and never counted.
- —Correct rewrites are proven by machine. If Bohrin can’t prove a rewrite is the same program, it doesn’t send it.
- —Every number has its margin of error. No score is shown without its sample size and range.
- —What wasn’t checked is said. A check that couldn’t run is “not measured”, never clean.
- —The rules are public. The specification behind every number is open, so anyone can recompute a result.
What it doesn’t do yet
- —Python graders and Python code today, starting with coding environments.
- —Graders that run an agent inside a container, and
bohrin auditfor a whole environment in one command, are coming at launch.