K8s in the Dark: Securely Packaging Your Stack with Zarf

Table of Contents

The 2 AM Sneakernet Reality

Picture this: It’s 2 AM in a windowless SCIF. Your phone is a brick, the server rack is screaming, and there is zero path to the public internet.

This is the ‘air-gap.’ Your mission is to deploy a 20-microservice stack onto a Kubernetes cluster that has never seen a Google DNS. Previously, this meant a nightmare of docker save commands, fragile shell scripts, and the constant fear of forgetting a single sub-dependency. If you missed one 50MB image, the whole deployment stalled, forcing a two-hour drive back to the office to fix the manifest.

Mastering these disconnected environments is what separates senior platform engineers from the rest. Moving from a ‘works on my machine’ setup to a fully sovereign cloud requires a different toolkit. That is where Zarf comes in. Developed by Defense Unicorns, Zarf is an open-source tool that bundles images, binaries, and Helm charts into a single, compressed .zst file. It effectively treats your entire application stack as a single, portable artifact.

Quick Start: Build Your First Package in 5 Minutes

You’ll need the Zarf CLI on a machine with internet access to pull the initial dependencies. Once the package is baked, you can carry it into your isolated environment on a hardware-encrypted drive.

1. Install the CLI

# On macOS/Linux using Homebrew
brew install zarf

# Or grab the Linux binary directly
curl -s https://api.github.com/repos/defenseunicorns/zarf/releases/latest | grep browser_download_url | cut -d '"' -f 4 | grep Linux_amd64 | xargs curl -L -o zarf
chmod +x zarf
sudo mv zarf /usr/local/bin/

2. Define Your zarf.yaml

The zarf.yaml file acts as your manifest. It tells the tool exactly what to scrape from the web before you go offline.

kind: ZarfPackageConfig
metadata:
  name: secure-nginx-demo
  description: "Air-gapped Nginx deployment"

components:
  - name: web-server
    required: true
    manifests:
      - name: nginx-deployment
        files:
          - nginx-deployment.yaml
    images:
      - nginx:1.25.3-alpine

3. Package and Ship

Run the create command to pull the images and wrap them up. A typical Nginx package usually clocks in around 20-30MB, but a full stack can easily reach 5GB or more.

# Create the package
zarf package create .

# Move the .zst file to your offline machine

# Initialize the cluster (sets up the internal registry)
zarf init

# Deploy the package
zarf package deploy zarf-package-secure-nginx-demo-amd64.tar.zst

How Zarf Solves the Registry Problem

Running zarf init does more than just install a binary. It deploys a local container registry and a Mutating Webhook inside your cluster. This is the core mechanism that makes offline deployments seamless.

In a standard environment, K8s nodes try to pull nginx:latest from Docker Hub. In an air-gapped site, that request times out after 30 seconds. Zarf’s Agent intercepts these ‘pull’ requests. It automatically redirects the node to the internal Zarf registry living on the cluster. You don’t have to manually rewrite 50 different YAML files to point to a local IP address. Zarf handles the translation on the fly.

State Management Offline

Zarf tracks exactly which component versions are running in your environment. If a deployment fails at 3 AM, you can run zarf package list. This gives you a clear view of your disconnected environment without digging through raw K8s logs or obscure manifests.

Handling Complex Helm Charts

Modern stacks rarely rely on a single Pod. Usually, you’re dealing with Helm charts containing dozens of dependencies. Zarf excels here by recursively identifying every image mentioned in a chart and pulling it into the bundle.

components:
  - name: database-layer
    charts:
      - name: postgresql
        url: https://charts.bitnami.com/bitnami
        version: 12.1.0
    images:
      - bitnami/postgresql:15.2.0-debian-11-r0
  
  - name: config-assets
    files:
      - source: ./scripts/init-db.sh
        target: /usr/local/bin/init-db.sh
        executable: true

In this scenario, Zarf fetches the Bitnami chart and the specific Postgres image. When you deploy this offline, Zarf ‘re-tags’ the Helm values. This ensures the cluster pulls the 800MB Postgres image from the local Zarf registry rather than trying to hit the internet.

Hard-Won Lessons from the Field

After years of wrestling with disconnected systems, I’ve learned that the smallest oversight can ruin a deployment. Here is how to avoid common pitfalls:

Trim the Fat: Container images are heavy. If you use ‘latest’ tags or bulky Ubuntu-based images, your .zst file will balloon. Switching from a standard Python image (900MB) to an Alpine-based one (50MB) saves significant transfer time.
Verify Your Hashes: Corruption is common when moving files via USB or cross-domain transfers. Zarf performs internal checks, but you should always run sha256sum on your package before starting the zarf package deploy process.
Modularize Your Components: Avoid putting your entire infrastructure into one massive component. Use the required: false flag for non-essential tools like Prometheus or Grafana. This allows you to skip resource-heavy monitoring if the local hardware is underpowered.
Lightweight Initialization: The zarf init command installs several services by default. If you are deploying to a tiny edge device with only 4GB of RAM, use the --components flag to skip the logging stack and save overhead.

Air-gapped deployments used to be a weekend-killing chore. By adopting a declarative tool like Zarf, you transform a manual, error-prone process into a repeatable, versioned workflow. Next time you find yourself in that silent server room, you’ll be glad you have one file that actually contains everything you need.