Red Hat routes 70% of AI inference to open-weight models

Seventy percent of Red Hat's AI inference traffic now runs on open-weight models, not premium frontier models. The company disclosed the figure as part of a pilot that puts a self-hosted open-weight model from China's Z.ai in front of as many as 1,400 engineers. The number is the latest evidence that model sticker prices tell CIOs almost nothing about total AI cost.

Red Hat did not ban expensive models. It set a $300 monthly allowance per engineer for externally hosted frontier models, with exceptions when extra spending is justified. That is $3,600 per engineer per year before exceptions — a concrete ceiling, not a mandate. Engineers choose the model that fits the task.

What did the price-per-token comparison miss?

Stephen Watt, distinguished engineer and vice president in Red Hat's office of the CTO, put the margin problem bluntly. Premium models are useful early on, but, he said: "after a while, you'd be crazy to stay on it because it just eats all your margin." Watt spoke to TechTarget's Patrick Thibodeau.

His claim survives scrutiny only partly. Red Hat did not publish its actual savings, its inference spend before the pilot, or what the Z.ai deployment costs to self-host. The 70% figure measures routing, not dollars saved.

Two other cases complicate the cheap-model math further:

• Xbow, a cybersecurity company, found in testing that some lower-priced models consumed substantially more tokens to solve the same problems than proprietary alternatives. The extra processing erased much of the expected savings.
• DataCamp, an online learning company, evaluated Google's open-weight Gemma model for its AI Tutor. Hosting providers could not consistently deliver the response times and capacity the application required. DataCamp is now switching from Anthropic to OpenAI, which it considers cheaper for its needs — not to an open-weight model at all.

The lesson cuts against the obvious one. Price per token matters, but so do tokens consumed, output quality, response time and the work needed to finish a task successfully.

What counts as an open-weight model versus a frontier model?

The categories overlap, and vendors exploit that ambiguity:

• Open-weight models publish their trained parameters. Organizations can download them and run them themselves or through a third-party provider.
• Frontier models offer some of the most advanced capabilities, often at higher cost.

A model can be both open-weight and frontier. The distinction is not a pricing tiers chart. It is a deployment decision.

Where does the rest of the money go?

The model bill is the small line item. Data platforms, metadata, governance and access controls carry the load, and AI raises the stakes on all of them.

Metadata is doing new work here. It once documented data; it now supplies the context and permissions AI systems need to act on information correctly. At Vanguard, metadata supports AI tool data discovery, agent permissions and traceability. It tells systems not only what to find but what they are allowed to do with it.

Sergio Gago, chief technology officer at Cloudera, described the failure mode in an October 30 interview with TechTarget's Eric Avidon. A prototype can look impressive until the team discovers its governance is inadequate for production — for example, an agent exposing one customer's data to another customer. Bolted-on safeguards may not fix a flaw built into the application.

"The governance of the data is a fundamental consideration that experimentation teams leave until the end, and then the use cases fail," Gago said.

Agents that interpret instructions and take action require those controls to be explicit and tied tightly to how the application works, according to the reporting.

What should CIOs actually budget for?

Some of these expenses belong to individual applications; others are shared platform and governance investments supporting multiple AI deployments. They never appear on a model's invoice, but they belong in the total-cost calculation. Gago offered no figures for that overhead, and neither did Red Hat, Xbow or DataCamp. The absence of hard cost data is itself a finding.

None of this argues against cheaper models. Red Hat's allowance model shows the sensible version: give engineers choices, cap the spend, demand justification for overruns. The trick is knowing when paying more is worth it — and no vendor's pricing page answers that.

Choosing the model is one decision. Running it, connecting it to business data and keeping it reliable and safe are separate decisions with separate budgets. The best choice is not the cheapest model, the most powerful one, or even the one that wins the pilot; it is the one that does the job at a reasonable overall cost without creating more problems than it solves.