Can AMD Finally Compete for Local AI Workloads?

There's a version of the local AI story that gets told constantly right now—privacy, sovereignty, cost control, the slow death of all-you-can-eat API plans. You've heard it. The more interesting question, which rarely gets a clean answer, is: on what, exactly?

The hardware conversation has been dominated by two players. Apple Silicon, which has developed a genuine cult following among developers who want capable local inference on laptops without a PhD in driver configuration. And NVIDIA, which has spent a decade making sure CUDA is so deeply embedded in the deep learning stack that alternatives feel like climbing a different mountain with worse gear. That leaves AMD as the perennial "what about them?" in any local AI hardware discussion—impressive specs, perennial software asterisks.

AI developer Sam Witteveen recently spent time with a full AMD workstation—a Threadripper 9980X CPU paired with a Radeon AI Pro R9700 GPU carrying 32GB of VRAM—and put it through a comprehensive workout: LLM inference, image generation, fine-tuning, and raw PyTorch training. The setup was provided by Zidex in partnership with AMD, which is worth noting upfront. Sponsored access doesn't make the results wrong, but it does mean you read the enthusiasm with calibrated skepticism.

The results, by his account, were genuinely surprising. That's worth taking seriously, because Witteveen is not an AMD booster by prior reputation—he's covered enough hardware to know what qualified praise looks like.

The Token Math That's Driving People Local

Before getting into the hardware specifics, Witteveen lays out the economic logic pushing developers toward local AI, and it's a sharper argument than the generic "privacy" framing usually gets.

"Tokens look like they're getting cheaper on paper," he says, "but practically people's bills are going up because agents and reasoning eat tokens at a completely different scale than chat ever did."

This is accurate and underappreciated. When you're running a coding agent or a multi-step reasoning pipeline—the kinds of tasks that represent actual productivity workflows rather than casual chatting—you're not consuming tokens linearly. You're consuming them exponentially. A single agentic task that involves tool calls, self-correction loops, and chain-of-thought can burn through what a month of casual chatting used to cost. Meanwhile, the AI labs are quietly walking back flat-rate pricing, focusing their all-you-can-eat plans on their own first-party products.

The second half of the equation: open-weight models have gotten genuinely good. Witteveen rattles off Qwen 3.6, Gemma, DeepSeek, Kimi as examples, and the characterization is fair—"these aren't toys." The gap between frontier models and their open-weight counterparts has narrowed from a chasm to something more like a few months of lag. For many real-world tasks, that gap doesn't matter.

Put these two things together and the case for local AI stops being ideological and starts being financial.

What 32GB of VRAM Actually Changes

The hardware spec that matters most here is the 32GB of VRAM on the Radeon AI Pro R9700. This is not the bottleneck where most GPU AI deployments break down.

Witteveen makes a point that often gets glossed over in GPU benchmarks: with 32GB available, you don't have to make the painful quantization compromises that define most consumer-grade AI experiences. Running Qwen 3.6 mixture-of-experts at a recommended 4-bit quantization? Fine. Smaller model? Go 8-bit. A model like Google's Gemma 4 in its 4B variant? Full resolution, no cuts.

The practical difference this makes isn't just about quality—it's about which models you can run at all, and how much you're compromising the one you chose. At 160 tokens per second on Qwen 3.6, Witteveen is reporting throughput that's not just usable for reading—it's fast enough for agentic pipelines where the model is serving as a backend rather than a conversational partner.

That said: these are self-reported numbers on a system provided by the vendor for promotional purposes. Independent benchmarks would be more persuasive. What we have here is a plausible data point, not a proof.

The ROCm Question, Revisited

If you spent any time in AI development five or six years ago, you have muscle memory around AMD GPU software support. The muscle memory says: impressive hardware, treacherous software ecosystem, stick with NVIDIA unless you enjoy pain.

Witteveen addresses this directly, and his credibility on the point comes from having lived through that era. "Honestly," he says, "this is the thing that 10 years ago when I was building a deep learning computer, made a lot of people kind of nervous. Back then, people would be seriously impressed by the AMD hardware, but the software compatibility was where the issue was. Now, I'm very happy to report this is just not an issue today."

The mechanics of what's changed: ROCm (Radeon Open Compute Platform) now ships as an integrated runtime inside LM Studio, so point, click, restart, and the application recognizes the GPU. Ollama works the same way. PyTorch has official ROCm wheels—meaning you go to the install page, pick ROCm from a dropdown, copy a pip command, and it installs. From there, "your existing PyTorch code mostly just runs fine out of the box."

Mostly. That word is doing real work in that sentence, and it's the honest word to use. ROCm's HIP translation layer handles the conversion from CUDA-native code, but "mostly" means there are edge cases, and edge cases have a way of showing up at inconvenient times. The documentation is actively maintained, which is a meaningful improvement from the state of things a few years back—but actively maintained documentation exists precisely because there are still rough edges.

For standard inference and fine-tuning workflows, Witteveen reports smooth sailing. Unsloth, the popular fine-tuning library, has published an official guide for AMD GPU use. ComfyUI, which has become the go-to tool for local image and video generation workflows, offers a ROCm installation option that got image generation running on the system without drama. Image generation speeds were described as "pretty quickly"—qualitative, but encouraging.

Linux Is Still the Real Answer

Here's the tension that runs through the whole video, stated clearly enough that it's worth holding onto: Windows support for ROCm is materially inferior to Linux support.

For consumer software like LM Studio and Ollama, Windows works fine. For anything deeper—PyTorch training, direct GPU programming, the kind of work where you're writing code rather than pointing at a UI—you want Linux. Witteveen demonstrates this by showing ROCm 7.2 on a Linux dual-boot, where he can run full PyTorch with GPU access, train a ResNet on CIFAR-10, load Gemma 4 via the Transformers library, and serve it through a Gradio interface. The same operations are constrained or unavailable under Windows.

This isn't a dealbreaker, but it's a real consideration. For developers comfortable with Linux, it's a minor friction point. For the broader audience that local AI tools are supposedly targeting—people who want privacy and cost control but aren't necessarily systems programmers—a dual-boot requirement adds meaningful setup overhead. Local AI hardware options aimed at that broader audience tend to abstract this away; AMD's current stack does not, at least not fully.

Who This Is Actually For

The Threadripper 9980X + Radeon AI Pro R9700 configuration Witteveen tested is not a consumer build. Threadripper is a workstation-class processor, and the Radeon AI Pro line is similarly positioned at professional and enterprise buyers. This is the kind of machine a development team might provision, or a serious independent developer might build as a dedicated AI workstation, not something you find in a Best Buy.

That positioning matters for how to read the review. The question isn't "should I buy this instead of a gaming GPU?"—it's "for serious local AI development work, does AMD now represent a viable alternative to NVIDIA?" On the evidence presented here, the answer appears to be yes, with Linux as the operating environment and an acknowledgment that the ecosystem is mature but not identical to CUDA's depth.

The local AI hardware landscape has genuinely fragmented in interesting ways—Apple Silicon for portable inference, NVIDIA for the deepest CUDA compatibility, and now AMD making a credible case for workstation-class deployments where VRAM capacity and raw compute matter more than ecosystem breadth. Whether the ROCm software story holds up under production workloads, at scale, run by people less expert than Witteveen—that's the question his video raises without quite answering.

The gap between "impressive demo on provided hardware" and "reliable production alternative" has historically been where AMD's AI ambitions have stalled. The hardware has caught up. Whether the software ecosystem has caught up durably, not just in the hands of an expert on a good day, is the thing worth watching.

---

Marcus Chen-Ramirez is a senior technology correspondent at Buzzrag. He can be reached at @MarcusCR.