local AI inference

10 stories tagged local AI inference.

NVIDIA's PAIR Turns Idle Home GPUs Into a Local AI Pool

NVIDIA's open-source Personal AI Router pools Ollama and LM Studio nodes across your network. What PAIR does well, and what it leaves unanswered.

Man in glasses holding dual GPU graphics cards against blue tech background with "TWO R9700 ONE STRIX" text overlay

Framework Desktop Gets Two R9700 GPUs via PCIe Bridge

Yuki Okonkwo2 weeks ago

AI. Yuki Okonkwo2 weeks ago

Framework Desktop Gets Two R9700 GPUs via PCIe Bridge

Level1Techs strapped two ASUS R9700 GPUs to a Framework Desktop using a Broadcom PLX bridge, hitting 192GB total memory for local AI workloads. Here's what worked.

Perplexity Open Sources Lily, a Local AI Engine for Apple Silicon

Dev Kapoor3 weeks ago

AI. Dev Kapoor3 weeks ago

Perplexity Open Sources Lily, a Local AI Engine for Apple Silicon

Perplexity has open-sourced Lily, a Rust-based inference engine for Apple M5 Max. Here's what the benchmarks mean and why the OSS move matters.

Comparison graphic showing llama.cpp declining versus FreeToken rising, with neon-styled boxes and "753B ON ONE GPU" text…

FreeToken vs llama.cpp: A Local AI Engine Reality Check

Bob Reynolds4 weeks ago

AI. Bob Reynolds4 weeks ago

FreeToken vs llama.cpp: A Local AI Engine Reality Check

UC Berkeley's FreeToken claims to run 753B parameter MoE models on a single GPU. Here's what the benchmarks actually show—and what they quietly obscure.

A sleek black network storage device with five front-access bays against a blue background, labeled with AMD Strix Halo NAS…

Minisforum N5 Max Review: NAS Meets AI Workstation

Bob Reynolds1 month ago

AI. Bob Reynolds1 month ago

Minisforum N5 Max Review: NAS Meets AI Workstation

The Minisforum N5 Max stuffs a five-bay NAS, AMD Ryzen AI Max+ 395, and dual 10GbE into one box. Here's what that actually means in practice.

Intel Arc Pro B70 graphics card displayed from the side, showing its blue exterior, yellow memory modules, and cooling fins…

Intel Arc Pro B70: 32GB of VRAM at a Real Price

Bob Reynolds2 months ago

AI. Bob Reynolds2 months ago

Intel Arc Pro B70: 32GB of VRAM at a Real Price

Intel's Arc Pro B70 offers 32GB of VRAM for $1,000 and free SR-IOV support. For homelabbers and AI tinkerers, it's a serious buy. Gamers should wait.

Diffusion Gemma Runs Locally—and That Changes Privacy

Rachel "Rach" Kovacs3 months ago

AI. Rachel "Rach" Kovacs3 months ago

Diffusion Gemma Runs Locally—and That Changes Privacy

Google's Diffusion Gemma runs on consumer GPUs at 700+ tokens/sec. For privacy, the real story isn't speed—it's that your prompts never leave your machine.

Man in tan shirt with confused expression gestures at text reading "WTF IS MTP??" and "2x faster tokens no tradeoffs??"…

Llama.cpp Gets MTP: Local AI Just Got Faster

Rachel "Rach" Kovacs4 months ago

AI. Rachel "Rach" Kovacs4 months ago

Llama.cpp Gets MTP: Local AI Just Got Faster

Llama.cpp just merged Multi-Token Prediction, giving local AI a ~25% speed boost. Here's why that matters for your privacy—and how to use it.

A person in headphones points at a Dell server surrounded by tech logos (NVIDIA, Hugging Face, Together AI) against…

Desktop AI Supercomputers: What Dell's GB10 Says About Tech

Samira Barnes6 months ago

AI. Samira Barnes6 months ago

Desktop AI Supercomputers: What Dell's GB10 Says About Tech

Dell's Pro Max with GB10 brings Nvidia's Blackwell chips to your desk. But who needs a 1 petaflop AI workstation at home, and what does it signal about computing's future?

Two men react dramatically next to an open PC case with multiple storage drives, with "BATTLE MATRIX" text overlay and…

Intel Arc Pro B60: Testing 96GB of AI VRAM for $5K

Tyler Nakamura7 months ago

AI. Tyler Nakamura7 months ago

Intel Arc Pro B60: Testing 96GB of AI VRAM for $5K

Level1Techs tests Intel's Battle Matrix with four Arc Pro B60 GPUs—96GB VRAM for the price of an RTX 5090. Real-world AI performance examined.