From Pilot to Production: Why the ROI Is in the Pairing

2 Minutes

I sat across from a CIO at a large financial services firm and asked him a simple quest...

I sat across from a CIO at a large financial services firm and asked him a simple question: how much has AI actually moved the needle this year? He rattled off six pilots, none of them in production, none of them being used by real people in real workflows. He's not alone – across many financial institutions, this is the standard answer right now.  

MIT's NANDA initiative looked at 300 enterprise AI deployments this year and found that only around 5% were delivering measurable financial return. Dig into why the other 95% stalled and it isn't one bad model. The frontier models help individual tasks get faster but the handover between steps, the coordination, the approvals, stays exactly the same. That’s where the bottleneck moves to.  

The Problem: AI Capability Without Enterprise Output 

Problem 1: Optimising the step, not the bottleneck 

Engineering teams may be 30–40% faster at writing code but ask them where that shows up in ROI and everyone shrugs. The reason is process structure: when you speed up one step, the queue simply moves to the next one. A developer who produces twice as many outputs still waits for requirements to be written, for business cases to be approved, for controls documentation to be signed off. The bottleneck shifts but does not disappear. 

The constraint was never the individual task. It is the coordination overhead between tasks: the handoffs, the review cycles, the waiting for inputs from the previous stage, the rework when context is lost in translation from one team to the next. That's where most of the time actually goes, and it's exactly the bit that a tool built to make one person faster never touches. 

Problem 2: No one watching the sprawl 

Every group in the business spins up its own agent because nothing's stopping them, and pretty soon nobody can tell you what's actually running, who signed off on it, or what it's costing. I've seen this sprawl: dozens of disconnected systems that never talk to each other, each one solving its own small problem while the organisation as a whole learns nothing. What one team figures out in January never reaches the team solving the same problem in March.  

Problem 3: Regulated industries cannot accept the risk 

The tools aren’t built for regulated industries. No history on the agent that made a decision. No audit trail. No human in the loop where one's needed. In a regulated business, you can't run something at scale if you can't explain what it did and why. So firms don't. The tool sits in a sandbox forever, technically working, never trusted with anything that matters. 


What needs to happen 

A more powerful model doesn’t fix any of the above on its own. Think of the models as the engine, but you need the operating layer, the car, to get moving. 

  1. It connects workflows end to end, so a requirements agent feeds straight into a business case agent, straight into delivery design, straight into compliance sign-off, and the handoff that used to take days happens in seconds. 

  1. It governs at enterprise level rather than the individual tool level, every agent versioned, permissioned and audited, and every output carrying a trail back to why it exists, so this stops being experimentation and actually becomes operational. 

  1. It lets the organisation learn from itself, so the solution to what one team figures out in January is sitting there ready for the team solving the same problem in March. 

Get that right, and the return on AI stops being a mystery. Pilots start making it into production, and the operating model that actually works is the one where humans and AI agents are doing the work together, not one replacing the other. 

The winners here won't be the firms with the flashiest model. They'll be the ones harnessing the right operating framework and governance as part of building the thing, not paperwork bolted on afterwards. Less exciting than a model release announcement, I know. But it's the version of this story where, next time someone asks that CIO how much AI moved the needle this year, the answer isn't six pilots and a shrug. 

london-skyline

Download

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Etiam sit amet est id augue molestie pharetra suscipit et enim. Donec maximus hendrerit augue, eu posuere eros venenatis eu.

Download Magazine

Site by Venn