Weakened tests, caught on the pull request

Tests are how you know code works. A test runs your code and checks the answer. If an agent is allowed to edit the tests, it can make them pass without fixing anything, and nothing in the result looks wrong.

The agent usually isn’t trying to trick you. It was asked to make the tests pass, and making a test check less is one way to get there. Here is what that looks like in practice:

1+@pytest.mark.skip
2 def test_parse_numbers():
3-    assert parse("1,2,3") == [1, 2, 3]
4+    assert parse("1,2,3") is not None

Before, the test checked that parse returned exactly [1, 2, 3]. After, it only checks that parse returned something. A broken parser passes that. And the new skip line means the test doesn’t run at all.

Other versions of the same move: a check wrapped in try: … except Exception: pass so its failure is swallowed, a tolerance loosened from 0.001 to 0.5, a function that did real work replaced with pass, or a test file deleted outright. In a pull request with hundreds of changed lines, these are easy to miss.

bohrin verify looks at what changed in your tests since your branch started, and lists every change that makes a test check less than it did before. Run it inside your repository, on the branch the agent worked on:

1pip install bohrin
2bohrin verify --since main

Here is the real output for the change above, where the agent also wrote “Fix parser, all tests pass” in its commit message:

• —Each fact is one change, in one plain sentence. S1 → W2 means the check went from strong (it checked the value) to weak (it checks almost nothing).
• —BGW-109 is the permanent name for this kind of change: “tests edited, deleted or skipped”. The same kind always gets the same name.
• —The commit messages that claim success are shown next to the facts, so you can see the claim and the change side by side.
• —Not checked says what verify doesn’t look at yet. It always tells you.

It reads the structure of the code, not the text. Reformatting a test, or moving it to another file or class, is not reported.

You don’t have to remember to run it. Add this file to your repository and GitHub runs verify on every pull request. Each fact then appears as a note on the exact line of code it is about, right where a reviewer is already looking.

1on: pull_request
2permissions:
3  contents: read
4  security-events: write
5jobs:
6  verify:
7    runs-on: ubuntu-latest
8    steps:
9      - uses: actions/checkout@v7
10        with:
11          fetch-depth: 0
12      - uses: prabhu-gopal/bohrin@main  # pin to a release tag or commit once you adopt it

• —fetch-depth: 0 is required. Bohrin needs your history to compare against.
• —Add strict: true to fail the check on any fact at all.
• —Add since: to choose the commit it compares from.

• —Tests deleted or emptied, and not moved anywhere else
• —Asserts removed: five checks in a test became two
• —A check made weaker: == [1, 2, 3] became is not None, or just assert result
• —Skips added: @pytest.mark.skip, xfail
• —Tolerances loosened: pytest.approx(x, abs=0.5) where it was 0.001
• —Timeouts raised: a test given longer to run
• —Failures swallowed: a check moved inside try: … except Exception: pass
• —A test hook planted: a conftest.py hook that can change test results
• —Test selection changed: pytest settings that change which tests run
• —Code replaced by a stub: a function that did work now just passes
• —Always-equal objects: an __eq__ that always returns True
• —Timers reassigned: time.perf_counter replaced, which can fake speed

It also lists softer observations that are worth knowing but often harmless: an expected value changed, a new test with no check in it yet, or the function under test newly mocked.

• —It reads Python code and pytest configuration. Other languages aren’t covered yet.
• —It doesn’t yet notice a test’s expected answer being copied into the code it tests.
• —It tells you what changed. It doesn’t tell you whether the code is now correct. That’s still your call.

It makes no network calls. It only reads your history and never runs anything from the repository, so you can point it at code you don’t trust.