Perplexity Open Sources Lily, a Local AI Engine

Perplexity released Lily on September 2, 2026, a local inference engine built in Rust and optimized for Apple's M5 Max chip, under open-source terms. The project targets a single model, Qwen3.6-35B-A3B, and squeezes performance out of Apple's Metal GPU API through custom kernels rather than relying on existing framework abstractions.

According to marktechpost.com, Lily achieves 1.23x prefill throughput and 1.35x decode throughput compared to MLX-LM on the same hardware. Those numbers are worth unpacking. Prefill is the phase where the model processes your input prompt; decode is the token-by-token generation phase you see as output. A 1.35x decode improvement is the one users actually feel: responses appear faster. The 1.23x prefill gain is more meaningful for developers building retrieval-augmented pipelines where large context chunks get processed repeatedly.

Gizmodo frames this as running AI on your GPU instead of the cloud, which captures the user-facing pitch accurately. The M5 Max has a unified memory architecture that collapses the traditional CPU/GPU boundary, and Lily's Metal kernels are written to exploit that directly rather than going through the MLX framework's generalization layer. Specialization costs breadth and gains speed; that trade-off is the whole project.

MLX-LM, Apple's own machine learning framework for Silicon, is the obvious comparison target. The M5 Max hardware improvements over M4 already pushed local AI benchmarks significantly, so Lily is compounding a hardware advantage with a software one. Whether beating MLX-LM by 23-35% in these specific conditions translates to a maintainable lead as Apple continues updating its framework remains unclear. Framework-level optimizations eventually catch up to hand-tuned kernels, unless the hand-tuning keeps moving.

The company is a cloud-based AI search product. Its business model depends on users querying its service, not running inference locally. Releasing a local inference engine doesn't directly cannibalize that, because Qwen3.6-35B-A3B running on a MacBook Pro and Perplexity's cloud product serve different use cases and user segments. But the framing of "reducing reliance on cloud infrastructure" as a feature cuts in a more interesting direction: Perplexity may be building toward a hybrid architecture where on-device processing handles lower-latency or privacy-sensitive tasks while the cloud handles more complex queries.

That framing, though, comes from the brief rather than from anything Perplexity has publicly committed to. The sources don't spell out a product roadmap. What they do show is a company with internal Metal/Rust expertise publishing that expertise as open source. The community benefit is real regardless of the corporate motivation.

From a community dynamics standpoint, single-model inference engines occupy an odd niche in the OSS AI ecosystem. Projects like llama.cpp earned broad adoption by supporting dozens of model architectures across every platform imaginable. Lily's scope is explicitly narrow: one model, one chip generation, one OS ecosystem. That narrowness is a feature for performance and a liability for community growth. Developers who need Qwen3.6-35B-A3B on M5 Max hardware get something highly optimized; everyone else gets source code they can learn from but probably won't run directly.

The Rust choice reinforces this read. Rust gives you memory safety without a garbage collector and predictable performance characteristics at the cost of a steeper learning curve and smaller contributor pool than C or Python. For a performance-critical inference kernel, that trade-off makes sense. For building a large contributor community, it narrows the funnel. Whether Lily accretes community contributions or stays primarily a Perplexity-maintained artifact depends heavily on whether the company invests in contributor documentation and broadens model support over time. The sources don't tell us anything about that yet.

The OSS AI inference space on Apple Silicon has become surprisingly competitive. oMLX delivers 47 tokens per second compared to LM Studio's 16 on Apple Silicon, though with reliability tradeoffs. MLX-LM is the framework-level foundation most of these tools build on or compare against. Lily's approach of bypassing MLX-LM's abstractions is a bet that hand-tuned Metal kernels for a specific model will stay ahead of generalist optimizations. That bet has historical precedent in GPU computing, where hand-tuned CUDA kernels regularly outperformed library abstractions until NVIDIA's libraries caught up. Apple's incentives to keep MLX-LM competitive are strong, which means Lily's performance advantage has a natural expiry date unless it keeps pace.

The labor dimension here is understated in the initial coverage. Writing custom Metal kernels for transformer inference is specialized work. Perplexity paid engineers to do it, and now that work is public. This is how a lot of productive OSS gets made. But it does mean the project's sustainability depends on Perplexity's continued investment, not on an organic contributor community that materialized around a shared need. If Perplexity pivots away from local inference work, Lily's development trajectory becomes uncertain fast.

For developers deciding whether to build on Lily today: the performance numbers are real on the specific hardware configuration tested, the Rust/Metal architecture is coherent, and the open-source terms mean you can read the code before committing to it. The risks are narrowness of scope, dependency on a single corporate maintainer, and the probability that MLX-LM or another actively maintained framework closes the gap. Whether that's an acceptable trade-off depends entirely on whether you need maximum performance on M5 Max hardware right now, or whether portability and long-term maintainability matter more.

Put differently: Lily is a sharp tool. The question is whether sharp and narrow serves your project better than broad and maintained by Apple.

Dev Kapoor covers open source software and developer communities for Buzzrag.