Green CI, Wrong Behaviour
The pull request looks great. Tidy functions, good names, twelve new tests, all passing. You approve it. Two weeks later a customer finds that invoices are rounding down, and when you dig in, the new test asserts the wrong value. The agent wrote the code, then wrote a test that confirmed what the code did. Both were consistent with each other, and both were wrong.
Agent-written code fails differently from a junior engineer's code, and your review habits were built for the junior engineer. As a rule of thumb, a person's mistakes usually look like mistakes: awkward code, a confused comment, a sign of struggle. An agent's mistakes are fluent. It writes wrong code in the same confident style as right code, so the cues you learned to rely on are gone. And there is a lot more of it.
- Your team merges agent code and you are not sure how closely it's being read
- Reviews feel like skimming because the volume went up
- Tests pass, and bugs still get through
- You are deciding what your review policy should be for agent-authored work
What to Look For
| Failure | What it looks like | How to catch it |
|---|---|---|
| Tests that prove nothing | Asserts what the code does, or mocks the thing under test | Read the tests first, against the spec |
| Invented dependencies | A package that doesn't exist, or one a squatter registered | Check every new dependency by hand |
| Scope creep | Unrelated refactors mixed into the change | Compare the diff with the spec |
| Reinvented helpers | A new utility that duplicates one you already have | Search before accepting new helpers |
| Silent failure handling | A broad catch that hides errors and returns a default | Look hard at every catch and fallback |
| Loosened checks | A test or lint rule weakened to make things pass | Flag any change to existing assertions |
Invented packages are worth a special mention. Models sometimes suggest package names that sound right and don't exist. Attackers register those names and wait, which is known as slopsquatting. The install succeeds, and CI stays green.
Change the Order You Read In
I read an agent's pull request in a different order from a colleague's, because the risk sits in different places.
- The spec or ticket. Know what should have happened before you see what did.
- The diff summary. Do the touched files make sense? Anything unexpected is your first question.
- The tests. Read these before the code. Would they fail if the feature were broken?
- New dependencies, config and migrations. These have the largest blast radius.
- Error handling and boundaries. This is where fluent code hides its mistakes.
- The rest of the code. By now, you know what you are looking for.
Make Tests Prove Something
The strongest check for a test is to break the code and see whether it fails. Mutation testing automates that. You can also do the cheap version: ask a fresh agent session, with no memory of the implementation, to write tests from the spec alone.
# Does the test suite notice when the code is wrong?
# First time only: npm init stryker (installs the test-runner plugin and writes a config)
npx stryker run --mutate "src/billing/**/*.ts"
# Every new dependency, for a human to check
git diff origin/main -- package.json pnpm-lock.yaml | grep '^+' | grep -v '^+++'
Keeping generation and evaluation separate matters, because an agent grading its own work is a poor judge. A second session or a different model reviewing the change has seen none of the reasoning that led to it, so it isn't anchored on it. That review is useful as a first pass, and it finds real things. It doesn't replace a person.
sequenceDiagram
participant S as Spec
participant G as Implementing session
participant T as Fresh test session
participant R as Fresh review session
participant H as Human reviewer
S->>G: Build this
S->>T: Write tests from the spec only
T->>G: Tests must pass
G->>R: Diff, with no reasoning attached
R-->>H: Findings
H->>H: Read the tests, risks and diff
Scale Review by Risk
| Tier | Examples | Review |
|---|---|---|
| High | Auth, payments, data migrations, infrastructure | Two people, line by line, and the design agreed first |
| Normal | Features and business logic | One person, in the order above |
| Low | Docs, mechanical renames, test additions | Automated checks and a quick read |
Encode this with path-based code owners, so that a change to the billing folder always gets the right reviewers. Don't try to detect who wrote the code. Hold every change to the same bar, and let risk set the depth.
What Goes Wrong
1. Treating green CI as a review
CI proves the tests pass. If the agent wrote the tests, that proves very little.
2. Reviewing huge diffs
Nobody reviews 3,000 lines well. Send it back to be split. The limit is the reviewer's attention, not the tool's speed.
3. Asking the agent if its code is correct
It will usually say yes, and it will sound convinced. Ask a fresh session to find problems instead.
4. Letting the author go missing
The person who opened the pull request owns it, and should be able to explain any line. “The agent wrote that” is not an answer in an incident review.
A Checklist for the Review Itself
When a large agent-written change arrives, I use a short list, because fluent code makes it easy to lose focus:
- Does this solve the problem in the ticket, or a slightly different one?
- Would each new test fail if the feature were broken?
- Is every new dependency real, maintained and needed?
- Are errors handled, or just caught and hidden?
- Does anything touch auth, money, or data deletion?
- Is there new code that duplicates something we already have?
- Could I debug this at 2 a.m.?
If the answer to the last question is no, send it back, even if everything else passes. Readability is a safety property. Code that nobody on the team understands is a liability that only looks finished.
Be kind to your reviewers as well. If you opened the pull request, read it yourself first, and note where you're unsure. A reviewer shouldn't be the first human to see the code.
What I Would Do First
Add a check that flags new dependencies and any change to existing test assertions, since those are the two cheapest catches. Set a size limit on pull requests. Then agree one rule as a team: whoever opens a pull request can explain every line of it, whoever or whatever wrote it.