The Real Cost of AI Isn't Training—It's What Comes After

The tech industry's fixation on training costs—the GPUs, the data, the energy—obscures where AI companies actually spend their money. According to IBM Technology's Cedric Clyburn, the real expense comes after training ends, during what engineers call inference: the phase when models are deployed and serving actual users.

This matters because inference costs scale with usage. Train a model once, deploy it to millions of users making billions of requests. The arithmetic is unforgiving.

Consider the Llama 4 Maverick, a 400-billion-parameter model released at FP16 precision. Running it at original weights requires 800 gigabytes of memory—five 80GB GPUs like NVIDIA's A100s, which retail around $10,000 each. That's a $50,000 hardware requirement before you've served a single user query. Now multiply that across data centers running 24/7 to meet demand.

"The majority of the cost around AI isn't during training but it's actually during the deployment and through a process that's known as inference," Clyburn explains. "This is actually where all the money is going to once a training job has finished."

The solution—quantization—sounds technical but the principle is straightforward: reduce the numerical precision of the billions of parameters that make up a model's weights. Instead of representing each parameter as a 16-bit floating point number (two bytes), compress it to an 8-bit or even 4-bit integer.

The Hardware Economics

Clyburn walks through the math using Llama 4 Scout, a 109-billion-parameter model. At its original BFLOAT16 precision, each parameter consumes two bytes. That's 220GB total, requiring three 80GB GPUs. Compress to INT8—one byte per parameter—and you're at 109GB across two GPUs. Push to INT4 and you're down to 55GB: one GPU, with room left over for the key-value cache and other operational overhead.

That's a two-thirds reduction in hardware requirements. When inference engines are rented by the hour or purchased by the hundreds, those savings compound quickly. And there's a performance benefit: "When we run this model, because it's a much smaller memory footprint, we can have up to a five times improvement on throughput," Clyburn notes.

The obvious question: what's the trade-off? Reducing precision should degrade accuracy. Red Hat's evaluation data suggests the degradation is minimal—less than 1% across benchmarks like AIME and GPQA. In some cases, quantization's regularization effect actually improves performance.

That evaluation sample size—half a million tests—provides statistical weight. But it's worth noting that these are standard benchmarks, not necessarily representative of every production use case. The 1% figure is an average; specific applications might see more or less degradation depending on task complexity and model architecture.

Matching Technique to Use Case

Quantization isn't one-size-fits-all. The video distinguishes between online and offline inference, each demanding different optimization strategies.

Online applications—chatbots, retrieval-augmented generation systems, coding assistants—prioritize latency. Users expect responses in seconds, not minutes. For these scenarios, weight-only quantization schemes like W8A16 (8-bit weights, 16-bit activations) work well because the GPU isn't continuously maxed out. Request patterns are bursty.

Offline inference—analyzing thousands of customer transcripts for sentiment, batch processing documents—keeps GPUs at full capacity. Here, formats like FP8 or INT8 accelerate computation rate because the bottleneck isn't latency but throughput.

The distinction reveals how deployment context shapes technical choices. The same model, serving different workloads, requires different compression strategies. This isn't purely technical optimization—it's economic optimization with technical constraints.

The Tooling Gap

Implementation relies on open-source infrastructure. Hugging Face hosts pre-quantized models from major labs. The LLM Compressor, part of the vLLM project, lets developers import models, apply quantization algorithms like SparseGPT or GPTQ, and deploy them on inference engines.

This ecosystem lowers barriers to entry, but it also creates dependencies. Organizations optimizing inference costs depend on these tools maintaining compatibility as models evolve. When Meta releases Llama 5 or Google ships Gemini 2.0, will existing quantization pipelines work seamlessly? The video doesn't address version compatibility or long-term maintenance.

There's also a question about who benefits most from these techniques. Startups without capital for massive GPU clusters gain access to frontier models. Hyperscalers already optimizing at scale might see marginal rather than transformational gains. The cost curve matters differently depending on where you sit on it.

What the Numbers Don't Tell You

The presentation is technically accurate as far as it goes, but it elides regulatory and policy implications. As jurisdictions develop AI governance frameworks—the EU AI /article/google-turboquant-ai-memory-crisis Act, proposed US federal legislation, state-level privacy laws—compliance costs could dwarf hardware savings. Model compression doesn't address data provenance, bias testing, or audit trail requirements.

Nor does it solve the fundamental scaling problem. Quantization makes existing models cheaper to run; it doesn't change the trajectory toward ever-larger models. If the industry response to efficiency gains is simply building bigger models, we're optimizing within a paradigm that might not be sustainable.

Still, for organizations deploying AI today under current regulatory regimes, compression techniques represent one of the few levers that directly impacts the bottom line. Reducing GPU requirements from three to one isn't incremental improvement—it's the difference between economically viable and economically impossible for many applications.

The question isn't whether to use quantization. It's whether the industry's focus on making large models cheaper distracts from asking whether large models are always the right solution.

Samira Okonkwo-Barnes is tech policy and regulation correspondent for Buzzrag.