A talk about implementing AI in an engineering org

Everybody Wants the Agent

Somebody has to build the floor.

Tjakoen Stolk · development manager, technical lead

• Land the title. Wait a beat before the second line.
• Set the promise: one roadmap, one ordering argument, and the parts that broke on me.
• "Nothing in this talk is about picking a model."

The thing we all actually want

An agent that opens pull requests while you sleep.

I want it too. It is the most interesting thing to happen to this job in my career.

• Be genuinely enthusiastic. Do not set it up as a straw man.
• Describe the loop out loud instead of listing it: you describe the ticket, it reads the codebase, writes the change, runs the tests, opens the PR.
• The talk only lands if the room believes you want the same thing they do.

And now the part nobody puts on a slide

Almost nobody wants to build the thing underneath it.

Unglamorous. Takes a quarter. Mostly consists of making a codebase legible to a machine that has never met you.

• Flat delivery. This is the thesis and it should sound like a fact, not a warning.

DORA, roughly five thousand technology professionals

AI amplifies whatever an organization already is.

Strong engineering organizations get faster. Weak ones fail faster.

• The most useful published sentence in the field. Say it once, slowly, then stop.
• Add the consequence out loud: right now it correlates with higher throughput AND lower stability, at the same time.
• Do not editorialise. The next slide does the work.

Faros AI · two years, 22,000 developers, 4,000 teams

Generation is elastic. Review is not.

• Arrow key walks adoption: 0, then 35, then 70, then 100. Do not touch the mouse.
• At 35 everything is still fine. That is the slide most rooms think they are on.
• At 70 the queue appears. At 100 say nothing for a beat and let them read the waiting count.
• "Review capacity never moved. It is a property of your team, not of the tool you bought."
• Then the numbers: median time in review up 442%, incidents per pull request up 243%, churn up 861%.

So here is the thing I would name

The substrate gap

The distance between what your tools can do and what your codebase can absorb.

The models are the same for everyone. This part is not.

• This is the handle. Everything after this refers back to it by name.
• "Nobody sells you a substrate. That is why it does not get bought."

There is a word for the thing you build

A harness.

The scaffolding that decides what a model sees, what it may touch, and who checks the result.

Not the editor you work in. That one you buy. This one you build, and it is the one that decides your outcome.

• Plant the term deliberately, and credit it: this is Anthropic's usage, and their two harness write-ups are where to send people who want more.
• Disambiguate in the same breath. Half the room uses "harness" for Claude Code or Cursor. That is the runtime, not the harness.
• "Everything after this slide is harness work."

The reference everybody points at

A thousand pull requests a week, no human-written code.

Running on standardized environments, dense internal docs and large test suites, built for humans, over years, before agents existed.

• Give the number first, let it impress, then take it apart.
• Stripe are open about where the agents fail: undocumented subsystems, and thin CI.
• "The reference implementation everyone points at is a substrate story wearing an agent costume."

Five stages. Eighteen months.

• Land at month zero, everything dashed. "This is the room. Nobody has started."
• Arrow to month 2: instrumenting is done. Then 6, then 12, then 18.
• Read the line under it each time. It says what should already hold and what is in flight.
• "Most organizations have bought the first bar and are hoping the last one arrives on its own."

The spine of the whole roadmap

A skill is written once, and then promoted.

You never build a second system. There is never a migration project.

• Walk the five stages out loud against the roadmap they just saw: by hand, scored, triggered, unattended, chained.
• "The command a developer runs today is the loop that runs unattended in six months. Same file."

And here is the whole thing that artifact belongs to

Three layers. All of them files.

• Read the three bands out loud: what it knows before it starts, what it knows how to do, what it can reach and what stops it.
• Land on the bar at the bottom. Every box above it is a file in the repository, reviewed like code, and that is the whole reason any of it survives a person leaving.
• "Nothing on this slide is a setting in somebody's editor. That is the point of it."

One epic, walked twice, the same model doing the typing

Repair after. Or specification before.

• Land on the bare path. Two rows hollow at the top, two outlined at the bottom, and hollow means a person.
• "Nobody chose this. It is what happens when the only thing you gave the agent was the prompt."
• Arrow once. The two outlined rows go away and three rows fill in, which is the same work done earlier by a file.
• "Same epic. Same model. The work moved from repair, which lands on a person, to specification, which lands on a file."

And here is where almost everybody goes wrong

Everybody builds the coder first.

It is the one you are least equipped to supervise on day one.

• Say why they do it: exciting, demos well, it is why the budget got approved. All fair.
• Then the turn, flat: "which is how you get the numbers I showed you."

So build them in this order instead

Supervision first. Coder last.

• Land on coder-first. Twelve changes, twelve landing on a human. Let them look at it.
• "This is what buying the exciting one first actually looks like on a Tuesday."
• Arrow once to flip it. Four lanes wake up, eight get caught, four reach a person.
• "Same coder. The difference is entirely what is standing in front of it."

Read the order downwards and it is one sentence

Each of the first four builds a piece of the supervision the fifth one needs.

Get those four good enough to defend in a code review, and the coder becomes an ordinary engineering problem.

• Name them against the figure still in their heads: plan gives it a spec, review a standard, docs a codebase it can read, QA a verdict that is not an opinion.
• "Build the coder first and you have bought a very fast way to generate work nobody has time to check."

Weeks, not months. No procurement conversation.

Mark every agent-authored change. This week.

It is the join key for every comparison worth making, and it cannot be backfilled.

• A label, a branch prefix, a bot author, commit trailers. Whatever your forge supports.
• Then the other two out loud: make one repository legible, extract two skills from people who do the work.
• "None of this is impressive. All of it is the foundation."

Six months in, the question you can finally answer

Is agent-authored change failure rate worse than human-authored?

Six months of marked changes is what makes that computable. Nothing else you report will land as hard.

• Set the scene first: skills in daily use, a catalog with a maintainer, the first triggers firing.
• This is also where the headless requirement pays off or bites. Same files, no rewrite.

A year in

Two or three loops, running on the most boring work you have.

Notice that none of them demo well. That is the selection criterion, not an apology.

• Name them: dependency upgrades with test verification, security findings off your existing scanner, flaky test triage.
• Human review before merge, always. Nobody is starting the work.
• Expect "that is not very ambitious." Agree, then say why boring is the point.

This is the stage everybody skips, because it has no demo

A skill nobody scored cannot be promoted.

• Start at zero. Nothing is promotable, and "it seems to be working well" is not a count.
• Arrow up. Release notes clears first, review comments next, and the third one is still nowhere near.
• "Same evidence. Three bars. The bar is what it costs when the skill is wrong."
• "Agree the number before the runs start. A threshold picked afterwards is just the number you already have."

One distinction, if I could only teach you one

An instruction is a request. A hook is enforcement.

They feel identical when you write them. They are not remotely the same object.

• Set this up, then immediately go to the confession on the next slide. Do not explain it abstractly.

Ten sessions, one written rule

An instruction is a request.

• Land on instruction mode. Four dashed cells. "These four ignored it, and they look exactly like the ones that did not."
• Arrow once to flip to the hook. Same four, struck through, zero violations.
• Then the confession: my standard requires a worktree per run. Written twice, published, linked from every repo. I ran the command that lists them. One entry. Never used one.
• "A standard that is written, published and never executed looks exactly like a standard that works."

Before an agent takes a task unattended

Do not ask the model whether it can do it.

Models are badly calibrated about their own capability. They will say yes.

• Score the substrate instead: files touched, coverage on those paths, is there a context file, are the criteria testable. Mostly computable without a model.
• "Amber means the coverage is missing. That is a fact about your repo, not a guess about difficulty."
• The percentage you can classify green, tracked monthly, beats any published benchmark because it is measured against your codebase.

Two boxes cost tokens.

• Land idle. Read the seven nodes across, and point out the two heavy borders: those are the model calls.
• Arrow once and let it run. It fails its tests, takes one fix, comes back, passes, opens a PR, lands on a person.
• "Two model calls. Five deterministic steps. That split is what makes a loop cheap and predictable at the same time."
• Worth stealing from Stripe, and available to a one-person project on day one.

The rule that decides whether any of this is safe

Contain at the environment boundary, not the action boundary.

Approval-prompt safety is neither safe nor fast. It trains people to click yes without reading.

• Name the four out loud: hard retry cap, disposable sandbox with real permissions inside it, scoped short-lived credentials, a tested kill switch.
• The click-yes line usually gets a rueful laugh. Everyone has done it.
• "Untested rollback is a plan, not a control."

Now the part I am most qualified to talk about

A broken check and a clean input produce identical output.

Silence.

I have shipped several of these. A column index off by one. A pipeline signal that never arrived. A regex that never matched anything.

• Let "Silence." sit on its own for a second.
• Every one of these was invisible until something measured it. This is the most dangerous mode, because the system reports success the whole time.

And it is not one failure mode, it is six wearing one coat

Six failures. Six identical reports.

• Land on the report column. Six systems, one week, and every line is something you would sign off on.
• "Nothing on this screen is false. That is the problem."
• Arrow once. Same marks, same rows, six different disasters underneath them.
• "Reading the report is not checking. That is the whole section in one sentence."

So the design rule

“

Every automated check must distinguish “I ran and found nothing” from “I did not run.”

Make each run emit what executed and what was skipped. A silent pass is not a pass.

• Short slide. Say it, let them write it down, move on.

And the one that cost me the most

It has never completed a single run.

Every layer above it is correct, and useless.

• Describe it first: layered credentials, multiple fallback paths, each layer documented, several individually verified. Elegant.
• Then the reveal. Then: "the lesson is not be careful. I WAS careful. Being rigorous everywhere else bought me nothing."
• Verify the leaf before you build the tree.

One number, if you only instrument one

Change failure rate, split agent against human.

Individual-level metrics never feed a performance review. Say it once in writing, then honour it.

• The others out loud: review latency, edit distance before merge, percentage of tickets classifiable green.
• Edit distance is the sneaky one. Merged as-is and rewritten-by-half look identical in every standard metric.
• The moment people suspect the numbers feed a review, you are measuring the gaming rather than the work.

Anthropic, on their own harness

Every component encodes an assumption about what the model can't do on its own.

They deleted their own sprint mechanism a model generation later. It was load-bearing, then it was dead weight.

• Name what in this talk is a bet: the triage classifier, the retry cap, the evaluator skills. All compensations for today's limits.
• Then name what is not: verification, documentation, containment, the marker. A better model makes those MORE valuable, not less.
• "Once a model generation, take a component out and see if the outcome degrades. Otherwise you are paying a tax for a problem that went away."
• This is also the answer to "should we just wait for better models". No: better models widen what a harness can reach.

If you take one slide away, take this one

Five things. None of them need a budget line.

1. Mark every agent-authored change. Today. It cannot be backfilled.
2. Publish ninety days of delivery history before anything changes.
3. Make one repository legible. Your busiest, with a senior in the room.
4. Extract two skills from people who do the work. Manual only.
5. Start the outcome log on the very first run.

If you can only do one this week, do the marker. Everything else can start in month three and still work.

• This is the slide people photograph. Slow down and let them.
• The first item is a deliberate reprise of the earlier slide. Say so: "you have seen this one already, and that is because it is the one that cannot wait."
• Say what is NOT on it: buying anything, picking a model, writing a policy. Those get easier once these five are behind you.
• The marker is the only irreversible one. Everything downstream is a join against it.

And here is why the dullest item on it is the urgent one

Four of these wait. One is a door closing.

• Start at this week. Every bar flat, nothing lost, and that is the only moment this picture looks like that.
• Arrow through month three, six, twelve. Four bars never move. Watch the top one.
• "A marker cannot be backfilled. Those months are not late, they are gone."
• "So if the meeting funds one thing, fund the boring one. It is the only item with an expiry date."

So, to close

The agents are already better than the codebases they are pointed at.

Everybody wants the agent. I would rather be the one who builds the floor.

The judgment is human. The typing, by design, is not.

• Paste the link as this slide lands.
• If nobody asks a question, ask them: "what would it take to mark every agent-authored change in your repos this week?"