FAST AND FURIOUS Why Hybrid AI Beats Burning Tokens

We clearly live in interesting, fast-moving times. I recently sat down with Derek Holt, CEO of Digital.ai, for an upcoming episode of his podcast, Released: The Story Behind the Software, to discuss the impact of AI on software development. Within days of our conversation, organizations had already picked up a new favorite hobby: saving tokens!

What happened? More and more vendors are moving to consumption-based licensing.

It feels a lot like the early days of cloud hype more than a decade ago. Remember? It felt like the sky was the limit, until the first invoice from our hyperscaler arrived. We shouldn’t be surprised by the price increases this time either. If you’ve followed the news about AI companies in recent months, it’s clear that the massive investments behind them will be passed on to us as users. In the past 12 months alone, AI companies have raised more than $225 billion through bond sales, and investors would like to see a return sooner rather than later.

So, is there a way out? We should have learned from the past. But as humans, when something as groundbreaking as AI comes along, we love to experiment and try out as much as we can. It’s a pattern that runs through human history.

The answer, though, is simple: THINK ABOUT A HYBRID MODEL!

What does that mean? Over the past decade, we could choose how to deploy our software development tools such as SaaS or on our own premises. With many vendors, AI has taken away that freedom. If you want the AI capabilities, SaaS is often the only option, and many vendors no longer let you choose the LLM either. The key question is whether we really need the largest LLM for the task at hand. How often do you generate code, and how often do you translate medieval poetry into another language? Large LLMs can do both, but at a much higher price than a smaller, optimized model of your choice that generates code, handles pull requests or improves the structure of task descriptions and requirements.

We’ve all seen prices for AI-ready servers skyrocket over the past 12 months. Even so, you may be surprised how quickly you can break even on self-hosted, or even air-gapped, models running on your own hardware. Simulations have shown that the three-year cost of SaaS-only solutions can be up to six to seven times higher than self-hosting, even after including the cost of buying and running the necessary hardware, and self-hosted setups sometimes deliver faster turnaround times. You also gain security, because your data never leaves your premises.

In agentic AI for software development, hybrid has two dimensions: the deployment model and the LLM. I believe we’ll see a mix of large and specialized LLMs in software development, with large models used as SaaS when you need to crunch petabytes of data, and specialized models handling day-to-day tasks.

Let me finish with one more thought. Many of the tools we see in the field have their roots in the pre-AI era and have been in place for years, sometimes decades. It’s time to think about modernizing them. Many organizations are unhappy with the results of their AI initiatives or are facing new security issues and infrastructure bottlenecks. One reason is a lack of context, caused by data siloed across tool stacks built around a best-of-breed approach. Making that context accessible is key to getting the most out of your tokens with agentic AI. At the same time, it’s just as important to upgrade your tool infrastructure to avoid bottlenecks in the CI/CD pipeline. The key word here is automation. If your tools are hard to automate, it’s time to think about a change.

For organizations ready to rethink their deployment models, LLM choices and automation across the SDLC, this is exactly where Digital.ai and knowmad mood can help. Derek and I dig into all of this in our upcoming podcast, Burning Tokens: The Hidden Cost of Getting AI Wrong, from what AI really costs to why humans need to stay in control. With our newly expanded partnership, the timing couldn’t be better.

Watch for the full conversation in the coming days.