The Check Ran Once in 165 Sessions, and I Got the Reason Wrong

One run in a hundred and sixty-five sessions

Here is the query I ran on a Wednesday night, three weeks after publishing a note about how well my workflow was going.

One hundred and sixty-five AI sessions in my largest client project since early July. One of them ran the check that is supposed to fire at the start of every single one.

Not one in ten. One.

I had built that check for a reason I still believe in, and I had written it up in I Was Shipping Faster Than I Could Understand It, where I said, in those words, that I had made every one of my repos run it. That sentence was published on the thirtieth of July. The number above is what it looked like on the nineteenth of August.

Across the four other repositories my table named, the picture was flatter still. Two hundred and eighty-eight sessions between them. One doctor run, the one above. Zero run reports, which are the short evidence notes a session is supposed to leave behind when it finishes. Meanwhile this site, where the whole thing was built, had eighty-seven sessions that ran it and sixty-two that wrote a report. All of those are a snapshot taken on the nineteenth of August rather than a standing count, and I will come back to why that matters more than it sounds.

The shape I read as a habit problem. It is a plumbing problem.

The story I told myself, and it was a good one

I want to be precise about what I did next, because the mistake is the whole note and it does not look like a mistake from the inside.

I did not shrug. I took the numbers seriously and started building an explanation, and the explanation I built was about discipline. It went like this. A check that fires when you sit down costs you something immediately: it interrupts, it hands you a list, it delays the thing you came to do. Whatever it buys you arrives later, and often it arrives for somebody else, which in my case means a future version of me who has forgotten why any of this is here. Machinery whose cost is now and whose payoff is later and elsewhere is machinery that gets skipped. Every time.

I liked that theory. I still like it, which is a warning sign I did not read at the time. It explained the other numbers on the same page. Only twenty-one of four hundred and fifty-four sessions handed off properly to the next one, and twenty of those twenty-one were in this repo, the one where I was watching. Fifty-nine sessions were named some variant of "Review pasted text," which is what it looks like when work arrives as a wall of text with no shape on it. The longest single session in the store ran to eleven thousand three hundred and ninety-six messages, which is not a session, it is a hostage situation.

All of that is true and none of it was the answer.

Then I ran it myself, from the wrong folder

Late in the day I did the thing I should have done first. I opened the client project and ran the check by hand.

It could not find its own program.

The check is a small shell script wired to fire at session start. Its job is to locate my tooling and ask it what is out of shape in this repository. Locating the tooling is three lines, and here they are as they stood:

if command -v pantry >/dev/null 2>&1; then
  pantry="pantry"
elif [ -f "../pantry/cli.ts" ]; then
  pantry="bun ../pantry/cli.ts"
elif [ -f "node_modules/@tjakoen/pantry/cli.ts" ]; then
  pantry="bun node_modules/@tjakoen/pantry/cli.ts"

Read it as a machine would. The first rung asks whether the program is installed globally. It is not, because I never published it anywhere. The second rung looks for a sibling folder next to whatever repo the session opened in, which resolves only if you are already standing inside the one directory where all my stack repos live side by side. The third rung looks for a vendored copy, which the client project has no reason to carry.

So from anywhere except that one directory, all three rungs miss, and the script prints a line saying the check did not run and exits quietly with a success code.

One hundred and sixty-five sessions in that repo were not one hundred and sixty-five sessions declining a check. They were one hundred and sixty-five sessions being told the check was unavailable, in a message small enough that neither I nor the machine reading it ever treated it as a fault.

The one recorded run was a human typing the command by hand. That human was me, and I did it from the directory where it happens to work.

Why the wrong answer was so comfortable

The reason I spent a day on the wrong diagnosis is not that I am careless with data. It is that the wrong diagnosis was a story about behavior, and I already had a moral ready for it.

A behavior story flatters you a little even when it accuses you. It says the tool is sound and the human is weak, which is a shape I know how to fix: more discipline, better nudges, a rule written down somewhere. A plumbing story says the tool never worked and you did not check. There is no moral in that. There is only a path with two dots in it.

Here is the part that generalizes past my particular script, and it is smaller and less satisfying than the theory I built:

A measurement tells you what did not happen. It will never tell you why, and the distance between those two is exactly the size of a confident wrong answer.

Every number in that first section was correct. The reading laid over the top of them was invented, and it survived three weeks because it was plausible and because nobody, including me, ran the one command that would have killed it in five seconds.

What I still cannot reproduce, including one I got wrong twice

Since the whole point here is not trusting figures you have not checked, I had this note's numbers re-derived from the store before writing it, and two things came back that I am obliged to print.

The first is that "two hundred and eighty-eight sessions" is exact arithmetic and I originally labelled it wrong. It is the sum of the four other repositories in my table. It is not every session outside this site, which is three hundred and six, because three small workspaces never made the table at all. So the confession has a confession inside it: while writing up how I misread my own measurement, I misread a different one, in the same document, on the same page.

The second is that the eighty-seven and the sixty-two are pinned to the nineteenth of August and have already moved. The store held four hundred and fifty-four sessions that night. It holds four hundred and sixty now. Every one of those six landed in this repository, and none of them was product work.

That last fact is the uncomfortable one, so let me put it plainly rather than bury it. On the nineteenth of August, fifty-three of the one hundred and forty-eight sessions in this repo were about the workflow, the standards, the tooling or an audit, rather than about anything a visitor to this site would ever see. Roughly a third of the work was the machinery, not the thing the machinery is for. Six sessions have landed since. Five of them were about the workflow.

Measuring the loop is loop work. The ratio got worse because I looked at it.

What it says now

The path is fixed. There is a fourth rung on that ladder that resolves by absolute path, so the script can find its tooling from anywhere on this machine, and it sits last on purpose so that a repo carrying its own copy still wins.

I pointed it at the client project this morning, and for the first time in a hundred and sixty-five sessions it answered:

21 checks, 0 failing, 3 due

A stale code graph, a missing end to end suite, and a context budget I have blown by seven thousand characters. None of it dramatic. All of it work that had been sitting there since early July, invisible, because the thing whose job was to say so could not open its mouth.

Eleven repositories now carry both the config and the instructions, and the check reaches all of them. That is where I have to stop, because that is a rollout and not a result. The thing I got wrong last time was calling a rollout a result, and I would rather not do it twice in one note.

So: a hypothesis with a date on it. If I run the same query in October and the four quiet repos are still quiet, then my discipline theory was right all along and I owe it an apology. If they are not, the check was simply never in the room. I do not know which yet, and the honest version of this note is the one that says so rather than the one that ties a bow on it.

What I do know is that for three weeks I had a number that looked exactly like an answer. One run in a hundred and sixty-five. It was never an answer. It was a symptom, and I read it as a verdict, and the difference cost me a day and a published sentence I now have to live with.

The check runs in that repo now. It took five seconds to find out it never had.

The judgment is human. The typing, by design, is not.