Do Agent Rules Make Better Code? I Ran a Blind A/B Test
I’ve been telling people that directory-scoped rule files make AI agents write better code. I maintain laravel-agent-rules, 49 CLAUDE.md files that sit next to every directory where Laravel makes a meaningful choice. I use it on every project.
I had never actually measured it. So I built a harness, gave Claude Code the same spec twice, once with the rules and once without, and had the results reviewed blind.
Both versions passed every test. The rules version had the better architecture.
The setup
One PRD: a multi-tenant inventory reservations JSON API. Organizations, users with admin and member roles, inventory items, and reservations that hold stock for 15 minutes before they’re confirmed, canceled, or expire. It has the things that separate careful code from code that merely passes: tenant isolation, a derived “available” quantity, row locking so two requests can’t both grab the last unit, and an expiry rule that has to hold even before the cleanup job runs.
Two arms, everything else identical:
- Control: a fresh Laravel 13 project with Pest and Sanctum. No agent instructions at all. Laravel 13’s skeleton ships its own
CLAUDE.mdandAGENTS.md; those were removed. - Rules: the same project plus
laravel-agent-rules, pinned to one commit and installed withapply-agent-rules.
Both got the PRD byte-for-byte on stdin, the same model (claude-opus-5-5), the same flags, a $40 budget cap, and a 2-hour timeout. The order was randomized; the rules arm happened to run first.
Testing the test harness first
Before running anything, I put the harness itself in front of a panel of models: Claude Code, Codex, and agy, each reviewing independently. They found 38 problems. A few would have quietly produced confident, wrong numbers:
- The hidden test fixture punished secure code. It created users with
User::create([... 'organization_id' => ..., 'role' => ...]). An app that correctly keeps those fields out of$fillablewould fail most of the suite. The arm more likely to do that is the one with rules telling it to. - Skipped tests counted as passes. Only
<failure>and<error>marked a test as failed. - The diff left out every new file.
git diff HEADdoesn’t include untracked files, which in a greenfield build is nearly all the code. - Tool-call counts were always zero. In Claude Code’s stream-json output, tool calls are nested inside assistant messages, not top-level events.
- The token comparison hid the cost of the rules. It compared
input_tokens, but rule-file context lands in cache tokens. In my smoke test,input_tokenswas 4. The real number was 87,437.
Each of those bugs would have produced a clean-looking number that was wrong.
I fixed all 38. The hidden suite went from 11 tests to 21, and I checked it against a reference implementation (21/21) and two deliberately planted bugs (both caught) before trusting it.
The results
| Control | Rules | |
|---|---|---|
| Hidden tests | 21/21 | 21/21 |
| Agent’s own tests | 88, all pass | 98, all pass |
| Cost | $1.39 | $3.72 |
| Turns | 22 | 75 |
| Total input tokens | 0.84M | 5.02M |
| Output tokens | 36.7k | 62.3k |
| Wall time | 5.1 min | 9.4 min |
| Diff | 35 files, +1,903 | 53 files, +2,352 |
Functionally it was a tie. Both arms built a working, tenant-safe API that passes every hidden test, including the ones for tenant isolation, the 15-minute expiry edge, and attempts to inject organization_id or status through the request body.
The rules arm cost 2.7× as much and took 3.4× as many turns. Most of the extra tokens are cache reads: the rule files get pulled into context and re-read on every turn. The trace shows the agent reading them deliberately, too: cat app/CLAUDE.md, then a loop over the rule directories before writing any code.
If pass rate were the whole story, the rules would be a bad deal.
The blind review
A hidden test suite tells you whether code works. It can’t tell you whether code is good. For that I built anonymized snapshots, X and Y, with every CLAUDE.md and the rules lockfile stripped out, and handed them to two fresh agents. They knew nothing about the experiment. They scored both snapshots against an 8-item rubric (0 to 2 each), independently.
| Rubric item | A: Control | A: Rules | B: Control | B: Rules |
|---|---|---|---|---|
| Authorization boundaries | 1 | 2 | 2 | 2 |
| Transactions and lock order | 2 | 2 | 2 | 2 |
| Cohesion | 2 | 1 | 2 | 1 |
| Error behavior | 1 | 2 | 1 | 2 |
| Database access | 1 | 2 | 2 | 2 |
| Test quality | 2 | 2 | 2 | 2 |
| Maintenance | 1 | 2 | 1 | 2 |
| Security hygiene | 2 | 2 | 2 | 2 |
| Total | 12 | 15 | 14 | 15 |
Both reviewers preferred the rules arm. One called it “clearly” better, the other “slightly.” Their reasons lined up, and I checked each one against the code.
Where the rules arm won
One place per business rule. The hardest rule in this spec is that an expired pending reservation no longer holds stock, even before the cleanup job marks it expired. The rules arm defines it once, as named query scopes with in-memory equivalents, plus a test that the two always agree. Control writes the expiry check out in six places across the model and the expire command. That’s the kind of duplication that becomes an oversell bug the first time someone updates five of the six.
Business logic that doesn’t depend on HTTP. The rules arm’s Actions throw their own exceptions with stable error codes (insufficient_stock, and so on), and they re-check authorization on the locked row themselves. Control’s services throw Symfony ConflictHttpException from six places and trust the controller to have authorized first. Call them from a queued job and you get HTTP exceptions with no authorization. Control’s 404s also leak the internal model class name: Laravel’s default No query results for model [App\Models\Reservation] reaches the client even with debug off.
An N+1 query nobody tested for. Control’s GET /inventory falls back to a fresh SUM query for every item without active holds. I measured it: 4 queries for 2 items, 12 for 10. Control even wrote an N+1 test, but every item in it had a hold, so it could never catch this. The rules arm has no fallback and turns on Model::shouldBeStrict().
Where control won
It’s leaner. Two cohesive services against nine Action classes. Two of those Actions, LockReservationAction and EnsureStockCommitmentsAction, are really private helpers dressed up as use cases. The rules arm also checks authorization two or three times per request: once in the route, once in the Form Request, once in the Action. Both reviewers docked it for cohesion.
Its tenant scoping is harder to bypass. Control scopes route model binding through a model trait, so any route on that model gets it. The rules arm binds by route parameter name, so a future route using {item} instead of {inventoryItem} would skip tenant scoping. It’s the most serious weakness either reviewer found in the rules arm.
Where it was a tie
Concurrency was a tie. Both arms lock the inventory row first, then the reservation, re-read state under the lock, and never lock in the opposite order. Neither had a cross-tenant leak or a way to oversell through the API. That’s the part I worried about most, and the model got it right without any help.
With the rules, the code was no more correct, but it was better organized, and it cost 2.7× as much.
What the rules actually changed
The structural differences line up with what the rules tell the agent to do:
| Indicator | Control | Rules |
|---|---|---|
| Action classes | 0 | 9 |
| Service classes | 2 | 0 |
| Form Requests | 5 | 6 |
| Row-lock call sites | 2 | 5 |
The rules arm used Actions, domain exceptions, query scopes, strict mode, and throttling, all of which the rules ask for. Control wrote solid default Laravel: services, Form Requests, and policies.
The ceremony is the honest cost. Some of what the rules produced is structure this spec doesn’t need. If I were shipping this code, I’d take the rules arm’s design, swap in control’s tenancy trait, and fold the two helper Actions back into private methods.
What this doesn’t prove
This is one pair. Agents are stochastic; a second control run might land on query scopes on its own. The reviewers are models too, though they had no context and no idea which snapshot was which. And the hidden suite topped out at 100% for both arms, so it couldn’t separate them at all. The architecture finding comes entirely from the blind review.
What I’m comfortable saying from one run:
- Agent rules don’t move pass rates on a well-specified task. A capable model with a clear spec already writes working code.
- They do move architecture, in the direction the rules point. Both blind reviewers saw it, independently.
- They cost real money. 2.7× here, nearly all of it rule context re-read every turn.
- Rules can overshoot. Helper Actions and triple authorization are what you get when “use Actions” is followed literally.
The fourth point is the useful one for me as a rules author. The next version of laravel-agent-rules will say when an Action is warranted, as well as how to write one.
Next
More pairs: at least five, maybe across a couple of different specs, with the same blind review on each. If “one place per rule” and “no HTTP in the domain layer” keep showing up on the rules side, it’s the rules. If they don’t, I got lucky once.
I’ll also harden the hidden suite with what this run taught me. A query-count test on /inventory with items that have no holds, and a check that 404 bodies don’t name model classes, would turn two of the reviewers’ findings into part of the score.
The harness (pinned versions, isolated config, randomized order, blind snapshots) is the part I’d encourage anyone to steal. This pair cost $5.11 in agent time.