We have all spent a bunch of time with ChatGPT or its equivalents and although magical in many ways, a couple of things stand out; firstly, why can’t it just say so when it doesn’t “know” rather than guessing and why do you get a different answer each time you ask the same question?
The reasons for this get to the heart of how AI works. In particular, its use of chance and stochastics explains why the value AI brings to software developers in Capital Markets will likely be quite small compared to other use-cases. Amazon.com’s ‘Recommendation Engine’ being a standout example, which is regularly cited as being responsible for 35% of Amazon’s revenue.
It gets hard to listen to all the unconstrained hype in Capital Markets. We have been here before with the hype cycle and ignoring the inconvenient truths about what it actually takes to create value with technology and deliver change in our industry will not serve us well.
For the record, I do not consider myself an AI sceptic, and you would need to be a hardened naysayer to not see the phenomenal potential for AI, building on the major successes it’s already had.
But when you look at this from a problem-oriented perspective and not from the point of view of the solution, you can begin to see why success in Capital Markets will be hard come by.
The “hammer & nail” analogy has been over-played, but I think it’s worth playing again here.
In Capital Markets application development, the problem (or nail) is to significantly reduce the cost of a large system build to solve a given business problem and the hammer (or solution) is some AI-based software engineering tool.
Looking first at the ‘nail’, Capital Markets problems typically have:
- A very low risk tolerance and an extremely high cost of failure.
- A high degree of uniqueness; a tendency to all be doing the same thing differently
- Poor data quality describing what roles individuals perform and their respective input/outputs.
- A constantly evolving business problem due to competition and regulation.
And for the solution, AI’s strengths are where there is:
- A very high-quality data set – "garbage in, garbage out” – data is the foundational fuel that determines model performance, accuracy, and reliability.
- A very low level of novelty, patterns repeat - AI operates best within its training data and struggles with unexpected, novel situations.
- A tolerable cost of failure - AI can confidently present false information as fact aka “hallucinations”
- No need to track the history of something (training on new tasks causes the AI to overwrite previous knowledge aka “catastrophic forgetting”)
At a high level it’s a complete mismatch. If AI is the ‘hammer’ then problems in Capital Markets are definitely not ‘nails’ whereas if you take the Amazon story, it’s a much better fit. (see Figure 1)
Figure 1 : Shows how 3 out of the 4 metrics are a poor fit for app dev in capital markets whereas 4 out of 4 are a good fit for the Amazon Recommendation Engine
If we look more closely at the actual work that needs to be done when building software, 4 of the 5 work items are really poor fits for AI applicability.
Gathering requirements, testing and deployment, onboarding/migration and architecture/design are all good examples of problems that AI will struggle with for the reasons mentioned above.
Only the 5th, writing high-quality code, has a claim to be suitable for AI. But writing code will likely only take up 25% of the overall effort, and 75% will be spread across the other 4. (see Figure 2)
Figure 2: Coding and implementation only accounts for one third of the workload.
For coding, if we look at AI tools that can provide code snippets and example code, there is a lot of good training data, very high repeatability in questions and no concerns about history but this is not the case if we look at AI tools that can provide ‘high-quality’ code on a highly complex code base that has evolved over many years. Which is what we need.
Writing high-quality code, rather than code that just works, includes writing code that is understandable by others, uses a simple way to solve a problem, is easy to debug and uses a consistent approach to the same problem across the code base.
Because AI operates on statistical probability rather than logical reasoning, it lacks a true understanding of the business logic, intent, or the broader, messy codebase. This results in "hallucinated", overly complex or inefficient solutions.
Good code is written with a complete mental model of the code base and the history of past decisions, events and constraints. AI suffers from ‘Catastrophic Forgetting’: When models are retrained or updated to learn new information, they frequently overwrite previously learned, valid knowledge, leading to a loss of past capabilities.
So, in essence, AI can produce code that will work (most of the time), but that is much harder to read, maintain, test and debug, making the effort to fix AI-generated code greatly outweigh the speed of generation.
Unfortunately, developer productivity is virtually impossible to measure, and the effects of bad code are often not visible until much later in the system’s lifecycle. So, expect the hype to be unconstrained for the foreseeable future and the adoption of AI code-generating tools to grow rapidly, with the enterprise value not growing at the same rate.
A similar historical example is the adoption of low-cost developers in the early 00’s, which looked phenomenal on paper but was equally hard to measure the results. The adoption spread like wildfire, but did anything actually get faster, cheaper, better? In software development, it certainly didn’t.
Unfortunately, the enterprise value of AI coding tools will not grow as fast partly due to coding only representing a quarter of the app dev workload and partly due to the significant unintended consequences.
Because of the reasons cited above, the AI development tools will add value as long as they are used by seasoned developers who know ‘what good looks like’ and know what the shortfalls are of the tools. The unintended consequences of these tools will be that they effectively ‘lower the bar’ for developers, allowing less skilled developers to rely on the tools without recognising the shortcomings. The amount of bad code and bad applications that this will create will erode much of the benefits.
Unlike other hype cycles like, say, blockchain, AI is genuinely impressive and seductive to the person on the street (or in the boardroom). This, and the need to find efficiencies, plus the fear of missing out, will make the enterprise pressure to be ‘using AI’ overwhelming. Just don’t be surprised if in 10 years, AI has been broadly “adopted” for software development in Capital Markets, but the cost of technology and the speed of change have not materially moved.
