Graders that can’t be passed without the work

Take a simple task: write add(a, b) that returns the sum. Here is a grader that looks reasonable. It loads the submission and calls the function:

1def my_grader(submission):
2    namespace = {}
3    try:
4        exec(submission, namespace)
5        namespace["add"](2, 3)
6        return True
7    except BaseException:
8        return False

It checks that the function runs. It never checks the answer. So a function that returns 0, or does nothing at all, passes. A model trained against this grader learns that doing nothing is enough.

Real graders are more careful than this, but they get fooled in the same basic ways. A submission can quit the program with a “success” code before any check runs. It can return an object that claims to be equal to every expected answer. It can edit the tests that judge it. This behaviour has a name: reward hacking. The model isn’t at fault. The grader has a gap.

Bohrin sends your grader submissions that are wrong on purpose. A good grader must reject all of them. Every one your grader accepts is a gap, and Bohrin proves it. It never needs your model, an API key or a GPU. It tests the thing that judges the model.

Step 1. Describe a task. A task is an ID, the prompt, and a reference solution you know is correct.

1from bohrin.ir.task import Task
2 
3reference = "def add(a: int, b: int) -> int:\n    total = a + b  # the sum\n    return total\n"
4task = Task(id="add", prompt="Write add(a, b) that returns the sum.", reference=reference)

Step 2. Get the wrong-on-purpose submissions. Bohrin builds them from your reference. Each keeps your function’s name and signature and does no work.

1from bohrin.mutate.battery import battery
2 
3result = battery(task)
4for candidate in result.candidates:
5    print(candidate.provenance.operator)

• —constant_implementation returns a constant of the right type: return 0
• —drop_side_effect has an empty body: pass
• —equality_spoofing returns an object that says it equals anything
• —raise_not_implemented raises NotImplementedError
• —success_exit exits with “success” before any check can run

Step 3. Run your grader on them, and score it. Bohrin never runs your grader. You do, in your own setup. Then Bohrin scores what happened. For the weak grader above:

Zero out of 100: it paid for an empty function. Now a grader that checks the value and its type, type(r) is int and r == 5, catches all five and scores 100. Checking the type matters. With == 5 alone, the object that claims to equal everything gets through.

Step 4. Check it still accepts correct work. A grader that rejects everything catches every cheat, so Bohrin also sends rewrites of your reference that it can prove do exactly the same thing, such as the same code with the comment removed. A good grader must accept them.

Step 5. Look up the fix. Every gap points at a named weakness, and every weakness comes with advice on how to close it:

• —Coverage Score: out of 100, the share of cheats your grader caught. Higher is better.
• —Correct-work acceptance: out of 100, the share of correct rewrites your grader accepted. Always shown beside the Coverage Score, because either number alone can be gamed by trading one for the other.
• —95% CI is the margin of error. One task is a small sample, so the range is wide (21–100). Add more tasks and it narrows.
• —not run lists every known kind of gap that applies to your grader and wasn’t tried yet. Not tried never counts as passed.

• —Python graders and Python code today, starting with coding environments.
• —Graders that take code, or run a program on inputs and compare outputs, are fully supported now.
• —Graders that apply a patch to a repository get template submissions today. Graders that run an agent inside a container are coming at launch.
• —bohrin audit, which runs the whole check on your environment in one command, is coming at launch. Today you run your grader yourself, as above.