Apple's Neural Engine: Specialized Chips vs. Data

There's a naming problem at the center of how people understand Apple's chips, and it's been quietly distorting conversations about AI hardware ever since.

"The name neural kind of makes people think that it runs neural networks or LLMs," says Dr. Brandt, founder of Kiraa, in a recent video breaking down Apple's silicon architecture. "But that's actually not what it's for. It can't run a model and it doesn't even know what an LLM is."

What the Neural Engine actually is, Brandt explains, is a dedicated matrix multiplication unit — a piece of silicon that does one thing at extraordinary speed. According to Apple's own M4 announcement, that's 38 trillion operations per second for the specific class of low-precision math it's designed for. The chip is rigid, non-programmable, and completely uninterested in anything outside its narrow specialty. You push numbers in; transformed numbers come out.

That's the whole trick.

The chef who only makes one dish

The framing Brandt reaches for is a specialist cook: someone who can only prepare a single meal but does it better than anyone alive because it's literally all they do. It's a good analogy, and it captures something real about how modern chip design works. The Neural Engine doesn't compete with the CPU or GPU for tasks — it relieves them. Developers can route specific calculations to the NPU (neural processing unit) and let the CPU and GPU focus on what they're actually built for.

Pair that division of labor with what XDA Developers describes as unified memory architecture — introduced with the M1, where CPU, GPU, and NPU all draw from the same memory pool rather than copying data back and forth between separate banks — and you get a system where each specialist component has immediate access to what it needs, with none of the latency overhead that traditionally bleeds performance from multi-chip setups.

This isn't new technology, exactly. According to Wikipedia, Apple's Neural Engine has been present in Apple silicon since the A11 Bionic in 2017. But the architecture has been steadily evolving, and what happened with the M5 represents a meaningful structural shift.

Nvidia's accidental empire

Before getting to the M5, there's a detour through the 1990s that's actually essential context.

In 1999, Nvidia shipped the GeForce 256 — a chip whose entire purpose was rendering game graphics without making teenagers' screens stutter. To draw a triangle, you take its corner coordinates and apply matrix multiplication to rotate it, scale it, project it onto a surface. Do that millions of times per frame, at 60 frames per second, and you need a very fast matrix multiplication engine attached to your triangle renderer.

What Nvidia built for gaming, in other words, was a massive parallel matrix multiplication machine with a specific UI problem attached to it.

The AI connection came years later, when three researchers at the University of Toronto — Geoffrey Hinton's group — used consumer-grade Nvidia graphics cards not to draw triangles but to train image recognition networks. The math was the same; the application was completely different. "Nvidia didn't set out to build an AI chip," Brandt notes. "They were just trying to draw triangles so kids could shoot things."

That accidental alignment between gaming hardware and machine learning math is what seeded the entire modern AI infrastructure industry. The reason data centers are stacked with Nvidia GPUs today traces directly back to a hardware design decision made to serve teenage gamers in the late 1990s.

What changed in the M5

When Apple first released the M-series chips, the CPU, GPU, and NPU were architecturally separate — distinct functional blocks on the die. With the M5, Apple embedded a neural accelerator inside every individual GPU core, integrating multiplication capability at a much more granular level.

Brandt describes the performance trajectory from building Kiraa's engine across successive Apple chip generations as "truly remarkable," noting visible uplift at each step. Real-world testing of the M5 Max local AI performance has borne that out in practice, with prompt processing benchmarks showing the M5 Max outperforming even Apple's own desktop M3 Ultra.

The architectural logic is sound: if the bottleneck in GPU workloads is often the matrix multiplication step, then embedding dedicated multiplication hardware inside every GPU core — rather than routing data out to a separate NPU block — removes that bottleneck at the source. Whether the specific performance gains match any particular figure is something Brandt and external benchmarkers will continue measuring, but the direction of travel is clear.

The cost per query question

This is where Brandt's argument shifts from hardware explainer to business case, and it's worth examining carefully because it carries an implicit assumption that deserves scrutiny.

His central claim: the industry has defaulted to a "bigger is better" model — more GPUs, more gigawatts, more data center square footage — when the actual question businesses should be asking is cost per query. What does it cost to generate one useful output? Not what does a GPU rack cost; what does a single inference cost, delivered at the quality and latency your use case actually requires?

"You don't need to be renting a stretch limousine to get to the shops when your bike will get you there for the fraction of the cost," Brandt says.

The analogy works for a certain class of workloads. Routine document summarization, classification tasks, local embeddings, on-device inference for standard business queries — for all of these, the hardware sitting on a developer's desk may genuinely be sufficient. The energy efficiency advantage of specialized low-power silicon is real, and it's a vector where Apple chips have measurable advantages over GPU clusters for inference tasks. The efficiency case for purpose-built AI chips isn't unique to Apple — Furiosa AI's Warboy chip has been making a similar argument about data center economics from a different angle.

But the framing has limits worth naming. "Most AI workloads" is doing a lot of work in Brandt's argument, and the boundary between "local is fine" and "you actually need the data center" is fuzzy and workload-specific. Training large models, running frontier-scale inference at high concurrency, fine-tuning on large datasets — these remain genuinely GPU-hungry operations that local Apple silicon can't replicate. The argument isn't wrong so much as it applies most cleanly to the inference end of a workflow that was trained somewhere much larger.

There's also a conflict worth flagging: Brandt is the founder of Kiraa, a company building software specifically optimized for Apple's chip architecture. His incentives point directly at you concluding that Apple Silicon is the right platform for your workload. That doesn't make his technical claims inaccurate — the chip architecture he's describing is real and verifiable — but it does mean his framing of "most businesses don't need a data center" deserves the same skepticism you'd apply to any vendor with skin in the game. The hardware argument and the sales argument are running in parallel throughout.

What actually needs to be built

The more interesting provocation in Brandt's video isn't the hardware explainer — it's the developer gap he identifies at the end. The chips have been getting measurably faster with each generation. The architectural improvements are real. The efficiency advantages over GPU clusters for inference workloads are documented.

And yet, software that actually exploits this hardware at the business layer largely doesn't exist yet. The development ecosystem built around Nvidia's CUDA framework — twenty-plus years of optimization, tooling, and institutional knowledge — has no equivalent for Apple Silicon. Developers know how to target Nvidia hardware because the entire industry trained on it.

Apple's unified memory architecture and Neural Engine have been available to developers since 2020, per XDA Developers' overview of Apple Silicon. The Neural Engine itself goes back to 2017. What's missing isn't the hardware; it's the software layer that translates hardware capability into deployable business solutions.

Whether that gap closes depends on developers choosing to build there — and whether the efficiency and cost arguments become compelling enough to pull investment away from the incumbent ecosystem. The chips are ready. The question is whether anyone shows up to use them.

Yuki Okonkwo covers AI and machine learning for Buzzrag.