BigQuery's Optimized Mode Cuts AI Costs by 94%

The cost problem in enterprise AI has a particular shape. It's not that large language models don't work on large datasets—they do, often impressively. It's that the economics of applying them row-by-row across millions of records can turn a promising data initiative into a budget conversation nobody wants to have. The standard workaround has been to use cheaper, less capable traditional ML techniques instead. The quality trade-off is real and widely understood. It just tends to lose out to the cost argument.

Google Cloud's newly announced optimized mode for BigQuery AI functions is a technical attempt to dissolve that trade-off rather than accept it. The mechanism it uses—model distillation combined with vector embeddings—is not new to machine learning research, but deploying it as a managed, largely automatic feature inside a data warehouse is a meaningful step toward making the approach accessible without requiring a team of ML engineers to pull it off.

It's worth understanding what's actually happening under the hood, because the headline numbers are striking enough to invite skepticism.

The Distillation Approach

BigQuery already offers AI functions—AI.IF and AI.CLASSIFY—that pipe data directly to a large language model. The default behavior is exactly what it sounds like: every row goes to the LLM, every row generates tokens, every row costs money. For 34,000 images, according to Google's demo presented by Rushabh Desai, that default approach consumed over 55 million tokens and took nearly 16 minutes to complete.

Optimized mode changes the fundamental architecture of what happens. Rather than sending all 34,000 rows to the LLM, BigQuery samples a subset of the data, sends that sample to the LLM for labeling, and then uses those labeled examples to train what Desai calls a "lightweight distilled model" locally within BigQuery's own compute infrastructure. That distilled model then handles the remaining rows—without touching the LLM at all.

"Instead of sending every row to the LLM," Desai explains in the video, "BigQuery takes a small sample of your data and asks the LLM to label it. It then trains a lightweight distilled model locally within BigQuery using your data embeddings."

The distilled model works because of the embeddings. Vector embeddings convert text, images, or other data into numerical representations that capture semantic meaning—so the distilled model isn't just pattern-matching on surface features, it's operating on a richer representation of what the data actually means. This is why, for tasks like classification or filtering (does this image show an obstacle? is this voice command asking the car to slow down?), the distilled model can generalize from a sample to the full dataset without collapsing into noise.

The result in Google's demo: 3 million tokens instead of 55 million. Execution time of just over 2 minutes instead of nearly 16. That's a 94% reduction in token consumption on a single query.

What This Requires—And What It Doesn't

The barrier to entry for optimized mode is deliberately low. Desai's demo shows the trigger mechanism as a single parameter addition to an existing query. If your table already contains embeddings—whether pre-computed or autonomously generated by BigQuery—the system auto-detects them and activates optimization without additional code.

"You can pass data embeddings as a function parameter," Desai notes, "or if you are using autonomous embedding generation, BigQuery will find them and optimize the job for you without any extra code."

This matters from an adoption standpoint. Features that require significant refactoring rarely get adopted at scale inside large organizations. Features that bolt onto existing workflows get used. Google is clearly aware of this, which is why the "one-line change" framing is central to their pitch.

That said, embeddings don't appear from nowhere. Organizations that haven't already integrated embedding generation into their data pipelines will need to do so before they can use optimized mode—or rely on BigQuery's autonomous embedding generation, which introduces its own questions about compute costs, latency, and control over how your data gets represented. Those aren't blockers, but they're not zero effort either.

The Economics Scale Nonlinearly

One detail in Desai's presentation deserves more attention than it gets in a four-minute video: the distilled model, once trained, persists and gets applied to future rows. The cost savings therefore compound with data volume.

"Because the distilled model is now trained, your savings grow with your data," Desai explains. "The more rows you process, the more you save in both time and tokens."

This is significant for the enterprise use cases Google is targeting—product review analysis, insurance claims processing, customer support classification. These aren't one-time queries. They're recurring pipelines that process new data continuously. A 94% token reduction that applies to every run of a pipeline, indefinitely, is a fundamentally different economics conversation than a one-time optimization win.

The obvious question is whether the distilled model's accuracy degrades over time as the underlying data distribution shifts—a common problem in production ML systems. Google doesn't address this directly in the demo, and it's the kind of question that sits in the gap between a compelling product demo and a production deployment. What triggers retraining? How does BigQuery communicate when the distilled model's performance has drifted below acceptable thresholds? These are answerable questions, but the answers live in the documentation rather than the pitch.

Classification and Filtering at Scale: The Right Use Case

It's worth being precise about what this approach is and isn't suited for. Desai is clear that optimized mode targets tasks like "classification, rating, or filtering"—and there's a real technical reason for that scope.

Distillation works when the LLM's reasoning on a sample can be meaningfully generalized to the rest of the dataset. For classification tasks—this image contains a pedestrian, this review is negative, this command is asking for a speed reduction—the pattern the LLM applies is learnable from examples. The distilled model is, in essence, learning the LLM's decision boundary on your specific data.

For tasks that require genuine reasoning about each individual row's unique content—summarizing a specific customer complaint, drafting a personalized response, extracting nuanced sentiment from an ambiguous review—distillation's generalization won't hold. The LLM's value in those cases is precisely its ability to engage with particularity. You can't distill that away and preserve it.

This is a real constraint, not a minor asterisk. Large organizations typically have both types of AI workloads: high-volume classification jobs where optimized mode will deliver exactly what Google promises, and lower-volume generative tasks where per-row LLM inference remains the only appropriate architecture. Knowing which workload category you're operating in is prerequisite to knowing whether this feature is relevant to you.

The Broader Context

Google is not the first to apply distillation techniques to the problem of LLM inference cost at scale—the research literature on knowledge distillation goes back years, and practitioners have been using embedding-based classification as a cheaper LLM proxy in production environments for some time. What BigQuery's optimized mode represents is the productization of that practice: removing the ML engineering overhead, integrating the workflow into a familiar SQL-based environment, and making the optimization largely automatic.

That productization is itself a policy-adjacent story. As AI analysis becomes accessible to organizations that couldn't previously afford it—smaller insurers, regional retailers, public sector data teams—the questions about what those organizations are doing with AI-driven classification at scale become more pressing. A 94% cost reduction doesn't just make existing AI workloads cheaper. It makes previously uneconomical AI workloads viable.

The infrastructure that makes mass AI analysis cheap and fast is also the infrastructure that makes mass AI analysis ubiquitous. Whether the organizations using it have thought carefully about what they're classifying, and why, and with what accountability mechanisms in place, is a question that sits well outside BigQuery Studio—but it's the question that scale inevitably raises.

---

Samira Okonkwo-Barnes is Buzzrag's Technology Policy and Regulation Correspondent.