How Optimization Powers Modern AI and Engineering

There's a concept sitting underneath almost every piece of modern technology you interact with—ChatGPT, your car's cruise control, the MRI that caught your neighbor's tumor early, that photo of a black hole that broke the internet in 2019. It's not neural networks. It's not big data. It's optimization. And most people who use these technologies every day have no idea it's there.

Steve Brunton, a professor at the University of Washington whose research sits at the intersection of machine learning, physics, and engineering, recently released a lecture kicking off a boot camp series on the topic. It's aimed at students and practitioners who want to actually understand the math—but the opening tour he gives of why optimization matters is worth parsing for anyone who thinks about AI and its applications.

His framing is deliberately expansive: "It's like asking what are the applications of calculus or statistics or linear algebra. Optimization is everywhere. It is ubiquitous in almost every modern industrial technology." That's not hype. It's an accurate description of the substrate.

The engine under machine learning's hood

Here's something that gets glossed over in a lot of AI coverage: when someone says a model was "trained," what they mean, mechanically, is that an optimization algorithm ran. The training process is an optimization problem. You have a model with some parameters (the knobs), you have data (the target), and you're trying to find the parameter values that minimize the gap between what your model predicts and what the data actually shows.

Brunton puts it plainly: "Optimization is the engine that powers machine learning. Data is the fuel."

This applies to everything from a basic linear regression (which has been around for centuries) all the way to the large language models running in browsers and apps today. The specific algorithms differ—stochastic gradient descent, Adam optimizer, and their variants are the workhorses of modern deep learning—but the underlying logic is the same: find the parameters that minimize a loss function (a measure of how wrong your model is).

What Brunton adds that's genuinely interesting from an engineering perspective is the question of physics-informed machine learning. The tension he's identifying is real: pure data-driven models don't inherently know that the physical world has rules. A fluid dynamics model trained only on data might produce predictions that technically minimize prediction error but violate conservation of energy. That's bad.

Two main approaches exist for baking physics in. The first is adding a penalty term to the loss function—essentially telling the optimizer "also penalize solutions that break this physical law." The second is harder: constrained optimization, where instead of discouraging bad solutions, you mathematically forbid them. Brunton walks through work by researcher Jean-Christophe Loiseau on fluid flows as a concrete example of the second approach—using what's called KKT (Karush-Kuhn-Tucker) conditions to enforce physical symmetries as hard constraints on the solution space. The result is a machine learning model that can't, by construction, produce physically incoherent outputs.

The broader insight here—that physics is largely a story about symmetries, and symmetries can be encoded as constraints in an optimization problem—is one of those things that sounds abstract until you realize it's the reason convolutional neural networks work so well on images. CNNs exploit translational symmetry: a cat in the top-left corner of a photo is still a cat in the bottom-right. That structure is built into the architecture, which is itself a form of constrained optimization.

Control theory: optimization with a clock

Control theory has a different flavor from machine learning, but Brunton frames it the same way: as a constrained optimization problem. You have a system that evolves over time (a rocket, a power grid, a car engine), you have actuators you can manipulate, and you have an objective you're trying to hit—land the rocket, maintain voltage, keep fuel consumption down. The constraints are the physics of the system itself.

What's interesting here is the historical layer Brunton drops almost offhandedly: "All of the robust control toolbox in Python and MATLAB is basically wrapped around Fortran codes that were written approximately the year I was born that solve linear matrix inequality optimization problems." The foundational algorithms for modern robotics and aerospace control are decades old. The current revolution is happening at the interface: machine learning is being used to build better models of system dynamics, and those better models are then fed into the same old optimization frameworks to produce better controllers.

Reinforcement learning—the approach behind systems like AlphaGo and robotics locomotion research—is essentially control theory with a learned model instead of a known one. The optimization is still happening. The novelty is in where the model comes from.

Inverse problems: running physics backwards

This is the section of Brunton's lecture that I find most conceptually interesting, partly because it's the least intuitive and partly because of how many things turn out to be inverse problems once you know the term.

The setup: you have a forward model (physics that describes how inputs produce outputs) and you have some observed outputs. The inverse problem is running that backwards—given the outputs, what were the inputs? That's the question MRI machines are answering every time they produce a scan. The raw data from an MRI isn't an image; it's a collection of indirect electromagnetic measurements. The image is reconstructed through an inverse transform. You're not seeing your knee—you're seeing the solution to an optimization problem.

Brunton's historical example here is a good one. When the Trinity nuclear test footage was published in a popular science magazine, the British physicist G.I. Taylor used nothing but four photographs—timestamps and a scale bar—to calculate the yield of the bomb. Four images. No classified data. Just dimensional analysis and physics run backwards. That's an inverse problem, and it's also a story about how much information is latent in data when you have the right model.

The same logic extends to imaging black holes, locating earthquakes with seismographs, and decomposing noisy video footage into a clean signal and a noise component. That last one—work Brunton attributes to researcher Isabelle Scherl—is almost absurdly satisfying: given a single movie of fluid flow, an optimization algorithm separates it into two movies, one with all the structure and one with all the noise. The inputs are scrambled together; optimization unscrambles them.

What makes inverse problems genuinely hard is that they're often ill-posed—meaning there are infinitely many inputs that could have produced the same output. A blurry photo doesn't have one "true" high-resolution original; it has a whole space of possibilities. Regularization (adding terms to the optimization that favor solutions with certain properties, like smoothness or sparseness) is how you make the problem solvable. Different regularization choices encode different assumptions about what the answer should look like, which is a design choice with real consequences.

Digital twins: the optimization feedback loop

All of this converges in the concept of the digital twin—a computational surrogate model of a real physical system. The idea is that running experiments on real hardware is expensive and slow. Running experiments on a well-calibrated simulation is cheap and fast. If you can build an accurate enough model of an aircraft, an engine, or a manufacturing process, you can do your optimization there before you ever touch the physical version.

Building the digital twin itself is a machine learning and optimization problem (you're fitting a model to whatever data you have). Using it is an inverse design problem (you're asking: what configuration of inputs produces the best outputs?). And if your optimization starts exploring regions of the design space where you have no data, that triggers active learning—a process of strategically collecting new data to fill the gaps. Which is, Brunton notes, also an optimization problem.

The recursive quality of this is a little dizzying: optimization to build the model, optimization to use the model, optimization to decide what data to collect to improve the model. It's turtles all the way down, except the turtles are actually useful.

Where Brunton is most direct about stakes is in the gap between current capabilities and what's still unsolved: "There is room to be improved in every single aspect of control theory, of machine learning, and of inverse problems." Coming from someone who works in this space, that's not a hedge—it's an invitation. The foundations are mature. The applications are not.

The question worth sitting with isn't whether optimization is important (it is, obviously, demonstrably). It's whether the people building systems on top of it—the product teams, the policymakers, the users—have enough visibility into the assumptions baked into their loss functions, their constraints, their regularization choices. Every optimization problem encodes a definition of "better." That definition came from somewhere.

---

Yuki Okonkwo is Buzzrag's AI & Machine Learning Correspondent.