Why Three Nodes Matter: The Real Cost of Proxmox HA

Christian Lempa's two-node Proxmox cluster worked fine until it didn't. The setup—complete with a Q device for quorum—handled basic clustering tasks without complaint. VMs ran. The cluster stayed up. For learning the basics of Proxmox clustering, it delivered.

But the moment Lempa tried anything resembling real high availability—live migrations, maintenance windows, actual failover—the limitations became impossible to ignore. So he added a third node, installed Ceph distributed storage, and documented what changes when you move from "technically a cluster" to "actually highly available."

The video lands at an interesting moment in homelab culture. More people are running production-adjacent infrastructure at home, but the gap between "works in my lab" and "works like enterprise infrastructure" remains wide. Lempa's project sits right in that gap.

The Problem Two Nodes Can't Solve

The Q device setup Lempa used previously is clever workaround architecture. In a two-node cluster, you need a third vote to avoid split-brain scenarios—situations where both nodes think they're in charge after a network partition. The Q device provides that vote without requiring a full third Proxmox node.

"Even if the cluster itself was high available, the VMs were not," Lempa explains, "because you cannot just magically start a VM on another node if its disk is only existing on the failed server."

That's the core issue. Cluster quorum solves the "who's in charge" problem. It doesn't solve the "where's the data" problem. Lempa's previous workaround involved NFS storage—an external server hosting the VM disks. But that creates a new single point of failure and introduces network latency for every disk operation.

Some viewers suggested ZFS replication as an alternative. Technically viable, Lempa acknowledges, but it introduces replication windows—those gaps between snapshot copies where you're guessing what the last successful sync captured. For databases or time-sensitive applications, that uncertainty matters.

What Ceph Actually Does

Ceph solves this by making the storage itself distributed. Instead of VM disks living on one node or an external NAS, they exist across all nodes simultaneously. In Lempa's three-node setup with a pool size of three, each VM disk maintains a copy on each node.

The practical result: live migrations that actually work. Lempa demonstrates moving a VM between nodes while it stays online—the ping never drops, services never pause. The system only needs to copy the VM's memory state, not the disk, because the disk already exists on the destination node.

"The virtual machine disc is no longer sitting on one single server or on an external storage," Lempa notes. "It is distributed across the entire cluster using Ceph and this is exactly the point where it becomes truly high available."

But Ceph's requirements are steep. Proxmox documentation recommends at least three identical servers with a minimum of 12 OSDs (Object Storage Daemons) distributed evenly—essentially four disks per node. Lempa compromised with one 2TB NVMe per node, which technically works but sits well below production recommendations.

Network requirements matter even more. Proxmox strongly suggests 10Gb networking minimum, ideally with separate interfaces for public Ceph traffic, backend replication, and cluster communication. Lempa used a single network interface for everything—another compromise that works in a homelab but wouldn't survive production load.

The Terminology Problem

One consistent thread through Lempa's walkthrough: Ceph's terminology actively works against understanding. Monitors, managers, OSDs, RBDs, placement groups, CRUSH rules—each term describes something genuinely necessary, but the naming conventions assume knowledge you're trying to acquire.

Monitors track cluster state and maintain quorum (familiar concept, new implementation). Managers provide metrics and management features but don't participate in quorum (less obvious why you need them). OSDs are the daemons that actually handle disk I/O. RBDs (RADOS Block Devices) present block storage to VMs. Placement groups organize how Ceph distributes data across OSDs.

Lempa walks through these concepts methodically, but even his explanations reveal how much domain knowledge you need before the pieces connect. For instance, understanding why he set placement groups to 32 instead of the default requires knowing that with only three OSDs and a pool size of three, every placement group needs all three OSDs anyway—more groups just add overhead.

This isn't a criticism of Lempa's explanation. It's an observation about Ceph itself: the learning curve is steep because the concepts genuinely are complex. Distributed storage systems can't be simplified past a certain point without lying about how they work.

The Cost Calculation Nobody Wants to Make

Lempa addresses this directly: "Buying servers, buying 10GB network interfaces and separate NVME drives, that is an expensive project."

He's right. Three servers with decent specs, 10Gb networking, NVMe storage—you're looking at significant money for homelab infrastructure. And those are just the hardware costs. The time investment to learn Ceph's operational model, troubleshoot issues, and maintain the cluster adds another layer.

The question becomes: what are you actually buying with that investment? If you're learning distributed storage systems professionally, or running homelab services that genuinely need high availability, the cost might make sense. But Lempa's careful about not overselling the value.

He explicitly distinguishes Ceph from backup solutions—a point worth emphasizing since distributed storage and backups solve different problems. Ceph protects against node failure. It doesn't protect against accidentally deleting a VM or corrupting data. You still need separate backup infrastructure.

This is where the Ugreen NAS sponsorship in the video actually serves the content. Lempa positions it as complementary infrastructure, not replacement—the NAS handles backups and personal data, Ceph handles high-availability VM storage. Different tools, different jobs.

Where Compromises Live

Lempa's implementation sits in that interesting space between "recommended configuration" and "what actually works." One OSD per node instead of four. Single network interface instead of three. These aren't mistakes—they're conscious tradeoffs based on budget, available hardware, and actual usage patterns.

The honest question is whether those compromises undermine the entire point of the project. If you're cutting corners on the foundational requirements, are you actually achieving high availability or just building a more complex system with similar failure modes?

Lempa doesn't answer this directly, partly because it depends on your specific use case. A homelab running a few services can probably survive on 1Gb networking. A setup hosting dozens of VMs with heavy disk I/O will hit bottlenecks quickly. The minimum requirements exist for reasons, but those reasons scale with load.

What's useful is seeing someone document the tradeoffs openly rather than either strictly following enterprise recommendations or pretending limitations don't exist. Homelab infrastructure lives in this middle space—good enough to learn from and useful enough to run real services, but not pretending to be datacenter-grade.

The three-node requirement, though? That one's non-negotiable. Lempa makes clear that Ceph doesn't really make sense below three nodes. You can find exotic configurations that technically work with fewer, but at that point you're fighting the tool rather than using it.

For anyone considering a similar project, Lempa's walkthrough provides something rarer than a how-to guide: an honest accounting of where the costs live, which corners can be cut, and which requirements exist for actual technical reasons rather than corporate-approved-architecture reasons. The distributed storage layer works. Whether it's worth the investment depends entirely on what you're trying to build.

Dev Kapoor covers open source software and developer communities for Buzzrag.