Beyond Tokens: Rethinking AI Total Cost of Ownership at Scale

IA FORUM MEMBER INSIGHTS: ARTICLE

By Laxmikant Pukale, Director, Advanced Capabilities - AI & Intelligent Automation, USAA

Every week brings another headline about the rising cost of AI. Some of them are genuinely startling. One large technology company reportedly exhausted its entire 2026 AI coding budget in four months. Gartner now predicts that by 2028, AI coding costs will overtake the average developer’s salary, driven by rising token consumption and the industry shift to consumption-based pricing.

The pattern is not isolated to one company. Through the first half of 2026, several large enterprises introduced per-employee monthly caps on AI coding tools, and at least one major technology division moved its engineers onto a cheaper alternative after token billing consumed an annual budget ahead of schedule. When organizations of that size are surprised by their own invoices, the problem is not carelessness. It is a cost model that behaves differently from the ones our budgeting processes were built for.

It is easy to read those numbers and conclude that AI has simply become too expensive. I do not think that is what is happening. I think we are measuring the wrong thing.

AI is not getting more expensive. AI is getting more ambitious, and more real.

Unit Prices Are Falling. Consumption Is Not.

The cost of inference has fallen at a remarkable pace. Stanford’s AI Index found that querying a model at GPT-3.5 level dropped from $20 per million tokens in November 2022 to $0.07 by October 2024, a more than 280-fold reduction in roughly eighteen months. Go back further and the curve is steeper still: the first commercial large language model API launched in 2020 at around $60 per million input tokens. Competition, better hardware, and model optimization keep pushing that number down.

So why are enterprise AI bills growing?

Because the workload changed. A simple chatbot query is one model call. An agentic workflow, where the model plans, calls tools, checks its own work, and iterates, consumes far more. Gartner’s analysis places agentic tasks at five to thirty times the tokens of a standard chatbot exchange. Add retrieval, which pushes thousands of pages of context into a single query, and the multiplier grows again. Work that cost a few cents as a one-shot call can cost dollars once it is rebuilt as an orchestrated agent session.

Token prices went down. Token consumption went up faster. That is the whole cost crisis in one sentence.

The Real TCO Starts After the Model

Most of us have moved past dead pilots. We got smarter about token math, and projecting model and cloud costs at the outset is now the easy part.

Scaling is where the real complexity shows up.

The gap is between what a pilot measures and what production actually costs. A pilot measures the model. Production pays for everything wrapped around the model, and most of that spend does not appear on the AI vendor’s invoice at all. It appears in engineering hours, data remediation, control design, and the slow accumulation of platform components that each looked reasonable in isolation.

In my experience, four things drive the cost sentiment as organizations try to scale AI.

1. Architecture complexity. AI largely rides on traditional IT architecture, but it exposes gaps that were previously invisible, particularly in data architecture and data quality. That is overwhelming for many organizations, and unavoidable if AI is going to deliver a real return. Layer on the new “must-have” components now on offer, such as gateways, observability, and evaluation frameworks, and the foundation bill grows before the first use case ships.

2. Use case integration. Every use case has its own workflow, and its own AI capability needs, which pulls business and technology teams into long solutioning cycles. Each new use case adds a risk and control layer, plus the cost of integrating with the existing ecosystem. The tokens look like pennies. The integration around them is where the return quietly disappears.

3. Human controls multiply touchpoints. Almost everyone can prompt now, but we are still some distance from understanding how to work with AI well. Human-in-the-loop design adds interaction points, and the governance and compliance layers behind those points, which most of us are still building, carry real and recurring cost.

4. Consumption pricing rewards usage, not outcomes. Per-token billing means heavy experimentation and productive work look identical on the invoice. There is no direct relationship between tokens consumed and productivity gained. Maximizing token usage is not a strategy. Rather than pushing teams to burn tokens, push them to produce better results.

Three Things That Help

1. Bring AI to your organization, not your organization to AI. Integrate AI incrementally into the architecture, governance, and vendor layers you already have, rather than reorganizing everything around AI. This sounds obvious, but the pressure to show AI results makes teams overlook it. Be careful about running change management at full throttle.

2. Do not over-rely on a single infrastructure model. Managed APIs and hyperscalers are the right starting point: no capital outlay and immediate access to frontier models. At scale, the economics shift. Industry total cost of ownership analyses suggests that self-hosted inference at healthy GPU utilization can run at a fraction of premium API pricing per million tokens. High-volume, predictable workloads increasingly justify hybrid or on-premises capacity, and keeping a portability path protects flexibility, data privacy, and negotiating position once the current subsidy era ends. The longer you wait to diversify, the more expensive diversifying becomes.

3. Build AI FinOps before you build more agents. Treat tokens as a variable cost of goods sold, not an IT line item. Meter consumption by use case, route simple requests to smaller and cheaper models, and use techniques such as semantic caching to remove redundant calls. The discipline is arriving quickly: the FinOps Foundation’s 2026 survey found that 98 percent of practitioners now actively manage AI spend, up from 31 percent two years earlier. In most organizations, though, it arrives shortly after the first surprising invoice rather than before it.

The Question Worth Asking

The organizations that struggle most with AI cost are usually not the ones spending the most. They are the ones that cannot explain what they bought.

A budget line that reads “tokens” tells a finance leader nothing. A line that reads “cost per resolved claim” or “cost per completed review” starts a very different conversation.

Measure cost per outcome rather than cost per token, and most of the panic in the headlines turns back into what it actually is: an ordinary engineering and finance problem, of the kind our organizations have solved many times before.

Author Disclaimer: The views and opinions expressed herein are those of the Author alone and are shared in a personal capacity, in accordance with the Chatham House Rule. They do not reflect the official views or positions of the Author’s employer, organization, or any affiliated entity.

References

1. Gartner, Inc., “Gartner Predicts AI Coding Costs Will Surpass Average Developer’s Salary by 2028 as Token Consumption Surges.”, Press release, June 24, 2026.

2. Fortune, “Uber burned through its entire 2026 AI budget in four months.”, May 26, 2026.

3. Stanford Institute for Human-Centered AI, “Artificial Intelligence Index Report 2025, Chapter 1: Research and Development.”

4. Gartner, Inc., “Analysis of agentic AI token consumption (five to thirty times a standard chatbot exchange per task)”, March 2026.

5. FinOps Foundation, “State of FinOps 2026”, (annual practitioner survey, 1,192 respondents).

6. Price-per-token market trackers and public model pricing pages, accessed July 2026.