A command is not an acceptance criterion

Last time I showed a bug that the tests had certified. The ticket said a trip longer than 100 km gets a surcharge. The code said >= 100. The tests, written by the same AI in the same session, agreed with the code.

The fix in that post was provenance: derive the test cases from the ticket, not from the implementation. But I skipped a question. What did the ticket actually say?

Here it is, the way most tickets look:

A trip longer than 100 km gets a 10% long-distance surcharge.

## Acceptance criteria
- [ ] Surcharge implemented in `Fare.Surcharge`
- [ ] Unit tests added
- [ ] `dotnet test` passes

All three boxes were ticked, honestly. The bug shipped anyway.

The criterion that was met

Look at what those three lines describe. A method to edit, a kind of file to add, a command to run. That’s a to-do list for the work, not a description of the result.

To-do lists are exactly what an AI agent is good at. It will implement the method, add the tests, run dotnet test and see green. Every criterion is satisfied, and the bug violates none of them.

A criterion that the broken code passes isn’t a criterion. It’s a formality.

There’s a second problem, and it takes longer to show up. Commands go stale.

I’m building an iOS app with a team of AI agents, and its issues used to carry verification commands in their criteria. One criterion said, in so many words: run the build with this flag.

A few weeks later the project’s rules said: don’t use that flag anymore. The issue still demanded it.

The agent that picked it up had two authorities that contradicted each other. Whichever one it followed, it broke the other.

A command describes how you check something today. Behaviour describes what must be true no matter how you check it. Only one of those survives a change to your build.

Bad and good, side by side

The rule I use now: a criterion describes something an observer can see, or evidence that must exist. A user, an API response, a test result. Never a command, a file name or a method name.

Here’s the same ticket, criterion by criterion:

The right-hand column has one property the left doesn’t: the >= bug fails it. Exactly 100 km is in there, in plain words, written by a human before any code existed.

That last row is the trap. Once you know the fix, it’s tempting to write the fix down as the criterion.

But “use >” is checked by reading the diff, and a diff can be right while the behaviour is still wrong somewhere else. “Exactly 100 km gets no surcharge” is checked by running the code.

Notice also what the right-hand column does for review. A reviewer can tick every line without opening Fare.cs. That’s what makes it a contract instead of a hint: you can verify it from the outside.

Red on the old code

The third criterion on the right is the one I’d keep if I could only keep one:

A test exists that fails on the current main.

A test that’s green on the old code and green on the new code proves nothing about the change. It might be a good test. It isn’t evidence that this issue got fixed.

The cheapest way to check it, before you commit, is to park the fix and keep the tests:

git stash push -- src/
dotnet test --filter "FullyQualifiedName~FareTests"
git stash pop

If that run is red, the test sees the bug. If it’s green, the test is decoration, however good it looks.

That snippet belongs in your workflow, not in the issue. The criterion says what must be true. How you prove it, with a stash, a cherry-pick or a CI job, is up to whoever does the work.

It’s also a cheaper cousin of the mutation testing from the previous post. Stryker proves your suite can fail at all. This proves that one specific test fails on one specific bug.

You don’t need a tool for it, only a sentence in the issue.

Contract versus direction

In the iOS app I mentioned earlier, every issue now has two sections with different weights.

Acceptance criteria are the contract. They’re binding, they describe behaviour or evidence, and the work isn’t done until each one holds.

Direction is a suggestion: “probably lives in Fare.Surcharge”, “check the boundary comparison”. The agent is free to ignore it if it finds a better route.

Before it starts, the agent reads both and compares them. If the direction contradicts the contract, the contract wins and the agent says so in the pull request. It doesn’t silently pick one.

The split isn’t bureaucracy. It gives the stale stuff somewhere to go.

Implementation hints age the moment someone refactors. Behaviour doesn’t. Keep them in separate sections, and a stale hint can be wrong without making the issue wrong.

This is a different job from planning mode. Planning is thinking before you code. This is writing down when you’re done, before anyone thinks about how.

Closing the loop

When the work is merged, the agent doesn’t close the issue with “Done ✅”. It posts a closing comment that ticks off each criterion with the evidence next to it. The test names come from the theory in the previous post. That’s the one written from the ticket instead of the code:

- [x] A trip of exactly 100 km gets no surcharge
      → Trips_longer_than_100km_get_a_10_percent_surcharge(100, 100, 0)
- [x] A trip of 100.1 km gets 10% of the base fare on top
      → same theory, row (100.1, 100, 10)
- [x] A test exists that fails on the current main
      → red on a1b2c3d with the fix stashed, green on e4f5a6b

“Done” is a bool. You can’t check it, only believe it.

A list of evidence per criterion is something a reviewer can follow, line by line. It’s the same idea as return the proof, not the answer, one level up: from a method signature to an issue.

It also gives the human review gate something concrete to do. The reviewer doesn’t have to reconstruct what the change was supposed to achieve. They hold the closing comment next to the criteria and look for the gap.

When not to do this

Spikes and exploration. If you don’t know yet what should be true, you can’t write it down. Write a question instead of criteria, and make the deliverable an answer.

Pure refactors. When behaviour must not change, “the existing tests stay green” is a perfectly good criterion. The behaviour is the existing suite. Just check that the suite covers the part you’re touching before you lean on it.

Trivial changes. A typo, a config value, a version bump. Writing three behavioural criteria for a one-line change costs more than it protects.

The pattern pays off where an agent picks up the work without you in the room. That’s exactly where “dotnet test passes” is most tempting, and least useful.

Rewrite one issue

Open your issue tracker and pick one open issue that someone is going to start on this week. Read its acceptance criteria and mark every line that contains a command, a file name or a method name.

Rewrite each marked line as something you could check without opening the code: what a user sees, what the API returns, which value comes out at the boundary. Then add one line that wasn’t there: a test exists that fails on the current main.

Paste the before and after of one criterion in the comments. You’ll have both in front of you the moment you’re done. I’m curious which commands turn out to be hiding in everyone’s definition of done.