Bohrin tells you whether your grader can be fooled, and proves it when it can.
A grader (a test suite, a reward function, a script that compares outputs) decides if AI-written code is right. If it can be passed without doing the work, a model learns to pass it that way. The model isn’t at fault. The grader has a gap.
1def add(a: int, b: int) -> int:2 return 0
1import sys23def add(a: int, b: int) -> int:4 sys.exit(0) # "success"
Ends the process with a success code before a single assertion runs.
1class Anything:2 def __eq__(self, other):3 return True45def add(a: int, b: int) -> int:6 return Anything()
Returns an object that claims to equal any expected answer.
1+@pytest.mark.skip2 def test_parse_numbers():3- assert parse("1,2,3") == [1, 2, 3]4+ assert parse("1,2,3") is not None
Weakens, skips or deletes the tests that judge it, then reports “all tests pass”.
76%
of attempts in which a frontier model cheated on coding tasks built so that passing is impossible without cheating.
Source · ImpossibleBench, 2025 ↗79%+
of those cheats, for several model families, were made by modifying the tests themselves.
Source · ImpossibleBench, 2025 ↗43×
more reward-hacking attempts by frontier models when the scoring code was visible to them.
Source · METR, 2025 ↗Bohrin checks the checker. It sends your grader submissions that are wrong on purpose. Every one it accepts is a gap, proven, with a script anyone can re-run.
Every known gapNo account, no model, no API key, no GPU. Runs offline, on your machine.
Compares what an agent changed with what it claimed. Weakened asserts, added skips and stubbed code, each stated as a plain fact.
Every score with its margin of error, and the smallest difference your evaluation can actually detect.
Sends your grader submissions that are wrong on purpose, and correct rewrites it must accept. Two numbers, each with its margin of error.
bohrin audit runs your grader on the cheats and the correct rewrites, streams findings as they’re proven, and prints the Coverage Score. The first finding should appear within about two minutes.
Every technique on the list was published by someone else first. Bohrin adds nothing secret to it.
Browse the listHollow programs
The grader pays for code that does no work.
Harness tampering
The grader lets the submission change what judges it.
Answer access
The grader lets the solution be found instead of produced.
Weak tests
The grader can’t tell a right answer from a wrong one.
Grader logic
The grader scores a failure as a success.
Isolation & permissions
The grader lets the submission reach things it shouldn’t.
Measurement adequacy
The grader is too small to support the claim.
Every finding ships with a script that runs your grader, unchanged. No Bohrin installed, no network. Run it yourself.
Only proven findings count. Anything less certain is shown for what it is, and never added to a score.
A grader that rejects correct code is scored too. A rewrite is sent only when a machine proves it does exactly what your reference does.
Every score with its margin of error, and the smallest difference your evaluation can actually detect.
16% of 445 benchmarks reviewed by 29 experts reported any uncertainty estimate or statistical test. Measuring what Matters, 2025
Before you ship an environment, find out whether its grader pays for answers that did none of the work.
219 flaws found across 10 agent benchmarks, most reaching near-perfect scores without solving a task. BenchJack, 2026
Your agent says “fixed, all tests pass”. Bohrin shows you whether it fixed the code or changed the tests.
76% of attempts in which a frontier model cheated on coding tasks built so that passing is impossible without cheating. ImpossibleBench, 2025
Every AI system has something that decides whether it did a good job: a grader, a benchmark, a test suite. Once machines did more work than people could check, that checker became the only thing between a model and a mistake. Nobody was checking the checker. So we built Bohrin to do exactly that.
Bohrin
Because AI deserves the truth
“Self-improvement is the first step toward greater artificial intelligence. We want AI systems to keep learning safe, correct information, so they can help humans go further.”PGPrabhu GopalFounder, Bohrin