Skip to main content

Lesleh.co.uk

Making robots argue about my pre-commit hooks

I’ve thought for a while that slow pre-commit hooks are pointless if CI runs the same checks anyway. If a pull request can’t merge until the checks pass, why make me wait over a minute every time I commit? I skip ours with --no-verify most of the time, which probably says enough.

I didn’t really want to argue about it with people, so I wrote a Claude Code skill that gets AI agents to argue about it for me. It’s called robot-wars. There are no flame pits, sadly.

CI runs it anyway. Citation needed. PRO CON JUDGE

How it works

You give it a motion, something a team could decide on, and it sets up two sides. PRO argues for the motion and CON argues against it. Both debaters get the same short context, a list of facts about the situation, and so does the judge.

PRO CON 400 words 400 words 300 words 300 words 300 words 300 words 150 words 150 words Openings written blind Rebuttal 1 Rebuttal 2 Closings Judge scores both sides and checks up to 3 facts Stress test tries to break the winning case Then it all runs again with the sides swapped. If the winner changes, it's a draw.

Each debater writes an opening without seeing the other one’s, then they take turns through two rounds of rebuttals and a closing. Every speech has a word limit. Every factual claim needs a source you can go and check, otherwise the debater has to label it as their own judgement. They can search the web for sources.

A judge then scores both sides on evidence, relevance, how well they answered each other, practicality and how fragile their case is. It checks up to three disputed facts itself, which turned out to be the best part. After that, another agent takes the winning case apart. It picks the premise most likely to be wrong, adds a fact the context doesn’t rule out, and checks whether the conclusion still holds.

The whole thing runs twice with the sides swapped. If PRO wins the first run and CON wins once the debaters swap sides, it’s a draw, because the speaking order decided it. That’s 20 agents for a full run, plus one each time a speech runs over its word limit and needs cutting down.

Round one went to CON

The first motion was more or less what I’d typed, “pre-commit hooks that take over a minute are pointless if every check is enforced as a required status check on pull requests in CI”. The context was two facts. The hook takes over a minute, and CI enforces every check it runs.

CON won both runs, narrowly the first time (36 to 35), and it won on a definition. CON took “pointless” to mean no value at all, so it only needed to find one benefit, and PRO never argued with that. CON’s benefits were that CI only builds the tip of a branch, so with merge commits or rebase-and-merge you can end up with untested commits on main, which makes git bisect less useful. And a secret you’ve pushed stays exposed even if the checks fail afterwards.

pointless adjective Having no value at all. CON

The judges caught both sides getting things confidently wrong too. CON kept saying the hook is the one point that sees every commit as it’s made. The judges pointed out that rebases and cherry-picks make new commits without running it, and that a plain hook checks your working tree, which can include changes you haven’t staged. PRO didn’t catch either.

The stress test broke CON’s case by assuming the repo only allows squash merging and the hook doesn’t scan for secrets. Both of CON’s arguments fall over at that point.

I also disagreed with a couple of things both sides took for granted. CI runs on every push, so it isn’t only checking the end of a pull request. And the hook does nothing about merges either, since it doesn’t run on rebased or cherry-picked commits. The robots didn’t know any of that because I hadn’t told them.

Round two, with the facts settled first

So the second time I settled the facts the first debate had turned on. Only squash merges. CI runs on every push, takes about two minutes, and nobody sits there waiting for it. People push often, but sometimes several commits at once. Some of us skip the hook because it’s slow. A fast secret scan can stay in the hook whichever side wins. AI agents commit too, and an agent can’t carry on until its commit finishes. And local environments aren’t clean, so local results can differ from CI.

I also reworded the motion to “pre-commit hooks that take over a minute per commit are not worth their cost when every check also runs in CI as a required status check”, so nobody could win on a dictionary this time.

PRO won both runs, 39 to 31 and 40 to 28, mostly because CON’s evidence didn’t survive the judges checking it. CON quoted a study on interruptions (Parnin and Rugaber) to show that coming back to a red build is expensive, but the study only counted breaks of 15 minutes or more, so CON withdrew it. It quoted the Claude Code docs saying “if something has to happen every time, make it a hook”, which sounds great until you read the rest of the sentence. That page is about Claude Code’s own hooks, and git doesn’t come into it. And the paper CON used for a 17.9% build breakage rate (Kerzazi, Khomh and Adams, 2014) names missing files as the most likely cause of a broken build, which is the one thing a check on your local working tree won’t catch.

WITHDRAWN Parnin and Rugaber interruptions study WRONG HOOKS Claude Code docs “make it a hook” BACKFIRED Kerzazi, Khomh and Adams 17.9% of builds break

PRO wasn’t perfect either. It claimed dismissed approvals cost the same with or without the hook, which doesn’t hold, since a red CI run and a fix push can dismiss approvals one extra time. CON didn’t spot it.

It’s still fragile

The stress tests broke PRO’s case in both runs, in different ways, and these are the bits I’d actually act on.

The first one added a failure rate. If the slow checks fail on one commit in four, and getting back to a red build costs you ten minutes, that’s about 2.5 minutes per commit without the hook against one minute with it. The hook wins.

1 min 2.5 min Average time lost per commit if the slow checks fail on 1 commit in 4, and each failure costs 10 minutes With the hook every commit waits Without the hook a quarter of 10 minutes

The second one was about agents. If an agent ends its session as soon as a push succeeds, without looking at CI, a broken commit just sits there until a person finds it. For that agent, the slow hook is the only feedback it ever gets.

So the answer is PRO, probably, as long as the checks don’t fail much and the agents wait for CI. That’s a lot more useful than the opinion I started with, because it tells me what to go and measure.

What I changed afterwards

The first debate was mostly wasted because the context left out the facts it turned on. So the skill now sketches each side’s strongest case before it starts, works out which facts that case needs, and asks me about them. It also words motions around the trade-off now, like “not worth its cost”. And the judge lists the arguments that survived on each side, with the assumption each one depends on.

A full run isn’t cheap. My two used 27 and 21 agents, about 900,000 tokens each, and took 8 and 18 minutes. I wouldn’t use it to pick a variable name. For an argument that’s been going round in circles for weeks I think it’s worth it, if only for the fact-checks, which catch the same confident nonsense people come out with in a real argument.

It’s still only the judge’s view of the arguments though. A well-argued side can be wrong, and I’m still skipping my hooks.