Running Kubernetes Air-Gapped: It's Still Hard in 2024

I've been watching people try to run Kubernetes without internet access since, well, since Kubernetes existed. The promise is always the same: total control, maximum security, compliance nirvana. The reality? You're signing up for a particular kind of operational pain that modern tooling can only partially alleviate.

Sidero Labs just released a 54-minute tutorial on deploying their Talos Omni stack in a completely air-gapped environment—no internet access, custom certificate authority, the full isolation treatment. It's thorough, it's honest about the complexity, and it's a useful window into why air-gapped infrastructure remains one of those problems that refuses to get easier despite decades of people trying.

The Architecture of Isolation

The video walks through the full topology of what you're actually building. You need Omni (the management plane), an image factory (to build your Talos images), a container registry (to serve everything), and authentication endpoints. All of these components need to talk to each other inside your isolated network, and none of them can phone home.

As the presenter explains: "All of the nodes need to be able to talk to Omni because that's where Kubernetes gets upgraded, Talos gets upgraded, patches get applied. All of that stuff happens through Omni."

Then you need an admin machine—something that bridges the gap, downloading components from the internet and transferring them into your isolated environment. The video demonstrates this with a home lab setup: one machine with internet access, another on a router with no external connectivity.

It's a reasonable architecture. It's also a lot of moving parts before you've deployed a single workload.

Certificate Hell, Revisited

The tutorial spends the first 13 minutes on certificates. That's not padding—that's how much ceremony is required when you're building your own certificate authority from scratch.

You generate a root CA. You add it to your trust store (differently on different OSes, naturally). You create wildcard certificates for your internal endpoints. You build certificate chains. You set file permissions so different containers running as different users can all read the same cert files.

This is table stakes. If you already have an internal CA, great—you still need to generate and distribute certificates for every endpoint in your stack. If you don't, you're learning certificate management whether you wanted to or not.

The presenter mentions Firefox maintaining its own root trust store, meaning you might still get certificate errors in the browser even after updating your system's trust store. These are the details that turn a two-hour project into a two-day project.

The Digest Problem

Here's where things get interesting in a "this is why air-gap is hard" way.

When you download container images to transfer into your isolated environment, you can't just grab them with any old tool and push them to your internal registry. Talos requires specific image digests—cryptographic hashes that identify exact image contents—and those digests need to match what's pinned in the Talos build.

"Trust me when I say you want to download these with something like crane that can repeat the same digest from an external registry to the internal registry," the presenter warns. "I promise you I've messed this up more than you have."

Container image tags are mutable—you can change what a tag points to. Digests are not. If you use a tool that downloads an image and re-packages it, you change the digest. Your builds fail. You spend hours debugging why your perfectly good images don't work.

This is not obvious. It's the kind of thing you learn by doing it wrong first.

Four Gigs Per Build

The video shows downloading all the images and system extensions for Talos. Each build is approximately four gigabytes. You're downloading everything—the base images, the system extensions for different hardware, the multi-architecture builds containing both ARM and x86 versions.

You can trim this down if you know you'll never need certain extensions. Don't run NVIDIA hardware? Delete those lines from your image list. But you need to make these decisions up front, because you're not pulling anything on-demand once you're inside the air gap.

This is the trade-off: complete control over what's in your environment, at the cost of complete responsibility for what's in your environment. No lazy loading. No "I'll grab that package when I need it." Everything you might need has to be there from the start.

The Human Element

What strikes me about this tutorial is how much of it is still manual ceremony. The presenter is clear about this: "We are going to do this manually. There are automated ways to do this, but we're going to show you every component so you understand how the pieces fit together."

That's good pedagogy. It's also a reminder that even with modern tooling, air-gapped deployment remains a specialist skill. You need to understand certificate chains, container image internals, network topology, and the specific requirements of every component in your stack.

The documentation has been rewritten three times, the presenter mentions. That's not a criticism—that's what good documentation requires. Air-gapped deployment is complex enough that even the people building the tools need multiple attempts to explain it clearly.

What Hasn't Changed

I remember similar conversations in the late 2000s about deploying software in isolated environments. Different tools, same fundamental problems: getting software in, managing updates, maintaining security, debugging when something breaks.

The tools have improved. Container registries are more mature than the package repositories we used back then. Certificate management has better tooling. The documentation is more comprehensive.

But you're still managing a parallel infrastructure. You're still making decisions about what to include before you know what you'll need. You're still debugging certificate chains and digest mismatches and networking problems without being able to just "try downloading it again."

Air-gapped infrastructure remains exactly what it's always been: maximum security and maximum operational overhead, chosen by organizations where the security requirements justify the cost. Talos and Omni make it more manageable than it used to be. They don't make it easy.

The question isn't whether you can run Kubernetes air-gapped—you can, and organizations do. The question is whether your security requirements actually need it, or whether you're opting into operational complexity because "air-gapped" sounds secure. Because if you're not sure you need this level of isolation, this 54-minute tutorial is a pretty good argument for not doing it.

—Mike Sullivan, Technology Correspondent