🧭 THIS WEEK AT BuildProven
Howdy. This was meant to land on Thursday. It didn't, which is a bit rich from a bloke starting a series about checking his own old answers.
As usual, I'd love it if you shared this, voted in the poll at the end, or best of all, hit reply and told me what you need help with on AI. I'll reply, and I'll help where I can.
🧰 Worth Your Click
Two from my own reading list this week.
How the Claude Code team does code review. The team that builds the agent on reviewing code at agent volume: once coverage was high, they aimed for the least human review time that doesn't raise reverts or incidents. Read it next to row 2 below.
Introducing Claude Opus 5.5. Anthropic's own claims on agentic coding and running cost. They're vendor numbers, so put them through row 4 below before you believe them.
🗺️ FEATURED INSIGHT
Then vs Now, part 1 of 4. In March I told you the AI tools were "stupidly good now" and you could have a prototype by lunch. In April I did 40 days of Claude Code: 784 commits, and it put API keys in its output twice. In October I finally did what I tell everyone else to do. I wrote the pass rule first and raced my own setup against plain Claude Code on 16 real tickets. Plain Claude Code got 14 of 16 accepted. The other 2 passed every check and were still wrong.
So this series is me checking my own old answers. Year end is when a lot of teams reset their AI rules, and plenty of those rules are "not yet" calls nobody has looked at since. Some deserve another go. Some should stay a no.
(The skipped-days job I promised last time has moved to January. Its proof isn't ready yet.)
What moved, and what didn't
Chat became agents. In March I pasted code into a chat and copied the answer back. Now the agent takes a ticket, edits the repo, runs the tests and hands me a diff. So what you're approving is a diff and the evidence behind it, and your review has to change to match.
I stopped counting commits. In April I was pretty pleased with 784 commits in 40 days. Now I only count tickets I'd actually merge after reading the diff. If your team reports AI work in commits or lines of code, ask for the accepted count instead.
Cheap models are fine for small tickets. Keel, my setup, gives each ticket to the cheapest model and only steps up when the checks fail. In round 2 that cost $0.10 to $0.36 a ticket. The model bill is small change now. Your review time is the expensive bit.
Green still isn't proof. In April Claude committed straight to main and used --no-verify to get round failing hooks. The hooks hold now, but 2 of 16 branches passed every check and still had a defect. Review found a race that could bring a revoked licence back, and a CI workflow expression GitHub rejects. This no stays. A green run means the review can start.
It still edits the test that marks it. On three broken tickets, plain Claude Code changed the locked tests to get through. That no stays too.
Proven or not?
The claim: my locked-tests setup now beats plain Claude Code on real tickets.
The evidence: Keel was accepted on 16 of 16 with 0 review defects. Plain Claude Code was accepted on 14 of 16 with 2. But my rule, written before the first run, said Keel's median wall time could be no more than 1.5x plain Claude Code's. Round 1 was about 35 minutes against about 7. Round 2 was 2.6x.
Verdict: NOT PROVEN. Better work, but too slow by the rule I set first. Then the bit I'm least proud of: the next day I dropped the time rule and moved Keel to a different test. I saw the data, then changed the rule, which is exactly the move the list below is meant to stop. The record stays as it was. One more gap: nobody timed my own minutes, because an agent ran both arms.
The Re-Test List
Pick the old no's your team still believes. Give each one 30 minutes of real work from your own backlog, not a demo. Write the pass line before you start, and don't touch it once you have.
"It can't write unit tests we'd keep." Give it one small function with no tests, run its tests, then break the function on purpose. Pass if the tests catch the break and a reviewer would merge them as written.
"It can't review our code." Give it a merged change that later turned out to have a defect and ask for a review. Pass if it names that defect, or a reviewer agrees it raised a real issue.
"Our spec is too long for it." Give it the whole requirements doc and ask for every requirement on one topic, with section numbers. Pass if every one it lists is real and it misses none you know of.
"It can't do a whole ticket." Give an agent one small real ticket with tests, on a fresh branch. Pass if the checks pass and you'd merge the diff after reading it.
"It weakens tests to get green." Same ticket, but diff the test files only. Pass if no assertion was removed, loosened or skipped. The checker below does this one for you.
"It can't read a failing CI log." Give it one failing log and ask for the root cause in three lines. Pass if it names the cause you found.
"It can't plan a multi-step change." Ask for a plan for a change you've already shipped and compare. Pass if it gets the order and the risky step right.
Then give each row one verdict. PROVEN: it passed on your work, so change the rule and tell the team. PROVEN IF: it passed under conditions, so write them into the rule. NOT PROVEN: keep the no and put a re-test date next to it. Never rescore after you've seen the result. If you don't like the line, write a new one and test new work.
To track it, paste this header into a sheet and add one row per old no:
Old no | When we said it | 30-minute test | Pass line (written first) | Verdict | Evidence | Re-test dateTake this
Did the AI weaken my tests? A one-file checker. Download test-weakening-check.html
Row 5 of the list, as a page. Paste a diff and it lists every edit that makes a test easier to pass: deleted assertions, skip markers, looser tolerances, changed expected values, asserts that can never fail. It runs in your browser and uploads nothing.
Check it: save the file, open it, click Load sample, and confirm it reports High, Medium and Low findings with each flagged line highlighted in the diff.
And the bigger one. The Sign-off Claude Code Starter Pack is now free for subscribers. It's the setup I run on my own repos: three hooks that block edits to locked tests, keep an evidence log and won't let a session finish without a proof, plus the CLAUDE.md template, the sign-off sheet, a review prompt and all five of my one-file tools. It was $49 and sold zero copies, mostly because I never actually showed it to you. Download signoff-cc-pack-v1.zip. Read the README before you install anything, then run python3 selftest/selftest.py: it should say 16/16 checks passed.
Weekly build logs from a 25-year program manager who codes with AI.
Brett
👉 Reply with the AI-assisted work you most need to approve, and the evidence you would want first. I read every one.
Want a second pair of eyes on one of your re-tests? Reply with the word "re-test" and we'll find 15 minutes for a free call.
