In reinforcement fine-tuning, you give the model a reward, a short piece of code that scores each answer, and training makes the model better at earning it. That is exactly the risk. If the reward can be earned without doing the task, you pay for a training run that teaches the shortcut.
The problem with a reward
A reward can go wrong in two directions. It can pay for wrong work, so the model learns a shortcut instead of the skill. Or it can refuse right work, so the model learns to avoid correct answers that happen to look different from the one you wrote.
Either way, you usually find out after the run: the scores went up and the model got worse. At training prices, that is an expensive way to learn your reward had a gap.
$100/hr
is what reinforcement fine-tuning is billed at on one major platform, up to $5,000 per job.
Source · OpenAI Help Center53.8–95.2%
is how often four widely used maths verifiers accepted correct rewrites of their own answers. The rest, they rejected.
Source · Where the Verifier Fails, 2026“Easily”
A developer fine-tuning with RL: keyword-based rewards “easily lead to reward hacking.”
Source · Unsloth issue 2690A reward that looks fine
Here is a reward for the task “write add(a, b)”. It runs the answer and gives full marks if nothing crashes:
1def my_grader(submission):2 namespace = {}3 try:4 exec(submission, namespace)5 namespace["add"](2, 3)6 return True7 except BaseException:8 return False
Bohrin sends it five answers that are wrong for certain, such as a function that just returns 0 or does nothing. This reward pays for them:
A model trained on this reward would learn that an empty function earns full marks. The fix is to check the answer, and its type:
1def good_grader(submission):2 namespace = {}3 try:4 exec(submission, namespace)5 r = namespace["add"](2, 3)6 return type(r) is int and r == 57 except BaseException:8 return False
Why the type check matters
With r == 5 alone, a submission can return an object that says it equals anything, and the reward pays for it. Checking type(r) is int closes that gap.
Checking both directions
A reward that rejects everything would catch every cheat, and would also teach the model nothing useful. So Bohrin always reports a second number beside the first: how often the reward accepts correct answers written a little differently, such as the same code with a comment removed or a local variable renamed.
Those rewrites are only sent if Bohrin can prove they are the same program. If it can’t prove it, it doesn’t send them. So when your reward rejects one, that is a real problem with the reward, not a judgement call.
- —Coverage Score: how many wrong answers the reward refused.
- —Correct-work acceptance: how many right answers it paid for.
- —Both come with a margin of error, and more tasks make it narrower.
What it doesn’t do yet
- —Today Bohrin tests rewards that judge Python code: ones that run the code, or run it on inputs and compare outputs. Other kinds of reward aren’t covered yet.
- —Bohrin never runs your reward itself. You run it, in your own setup, and Bohrin scores what happened.
- —It tests the reward, not the model. It needs no model, no API key and no GPU.