Posted inDevOps

Deploying Talos Linux for Kubernetes: Secure and Minimal Immutable OS

DevOps tutorial - IT technology blog
DevOps tutorial - IT technology blog

Context & Why

It is 2:15 AM on a Sunday. PagerDuty fires a Sev-1 alert: three worker nodes in the production cluster are reporting NodeNotReady. You SSH in to check the logs, only to discover that an engineer ran a manual apt-get upgrade during Friday’s shift. That update bumped a shared glibc dependency and broke the kubelet runtime. Configuration drifted silently, and nobody noticed until the scheduler attempted to roll out a routine deployment.

General-purpose distributions like Ubuntu, Debian, or RHEL were never built solely to run containers. They bundle package managers, systemd units, interactive shells, user accounts, and thousands of dynamic dependencies. Every mutable component on that disk is both an attack vector and an operational hazard.

Talos Linux discards this legacy model entirely. It is a purpose-built, immutable operating system designed exclusively for Kubernetes:

  • No SSH or Shell: There is no bash, no Python interpreter, and no SSH daemon listening on port 22. You cannot log in interactively or hot-patch running binaries.
  • Read-Only SquashFS Root: The core OS boots into a read-only filesystem in RAM. Ephemeral state stays isolated, preventing malware persistence and unauthorized disk writes.
  • API-Driven Management: Every operational task—from network setup to kernel sysctl tuning—runs over a secure gRPC API on port 50000 using talosctl and mutual TLS (mTLS).
  • Tiny Footprint: The entire installation image is roughly 80 MB, compared to 1.5–2.5 GB for standard enterprise Linux distributions. It packages only the Linux kernel, a custom Go-based PID 1 init system, containerd, and core Kubernetes components.

Adopting an immutable OS eliminates configuration drift at the root. When a node misbehaves, you no longer spend hours debugging broken system packages. You simply wipe the node or roll out a pristine image.

Installation

You manage a Talos cluster declaratively from your local workstation using two binaries: talosctl for OS management and kubectl for workload orchestration.

1. Install talosctl and Prerequisites

Download the official talosctl client:

# Download and install the latest talosctl binary
curl -sL https://talos.dev/install | bash

# Verify client version
talosctl version --client

2. Generate Cluster Configuration Files

Talos replaces manual installation wizards with declarative YAML configurations. Generate cryptographically signed machine configs for both control plane and worker nodes upfront.

In this example, the control plane endpoint resolves to 192.168.1.100:6443, the worker sits at 192.168.1.101, and the cluster name is talos-prod-01:

# Generate PKI secrets and machine configurations
talosctl gen config talos-prod-01 https://192.168.1.100:6443 \
  --output-dir ./_out

The command outputs four critical files inside ./_out:

  • controlplane.yaml: Machine config for control plane instances (runs etcd, kube-apiserver, kube-controller-manager).
  • worker.yaml: Machine config for compute nodes running container workloads.
  • talosconfig: Admin credentials containing client certificates for talosctl.
  • secrets.yaml: Root certificates, encryption keys, and bootstrap tokens.

3. Boot Nodes with the Talos Linux Image

Boot your target hardware, Proxmox VM, or cloud instance (AWS AMI, GCP, Hetzner) using the official Talos ISO or raw disk image. On initial boot, Talos enters maintenance mode, exposes its gRPC API on port 50000, and waits for configuration data.

Configuration

Once your instances are reachable on the network, push the generated configurations directly over gRPC.

1. Configure the Local talosctl Context

Point your workstation CLI to the generated talosconfig and define the target node endpoints:

# Point talosctl to the generated configuration
export TALOSCONFIG="./_out/talosconfig"

# Set default target endpoint and node
talosctl config endpoint 192.168.1.100
talosctl config node 192.168.1.100

2. Apply Machine Configuration to Nodes

Send the YAML definitions to each machine. Talos validates the schema, installs the OS onto the target drive (e.g., /dev/sda or /dev/nvme0n1), and triggers a reboot.

# Apply config to the Control Plane node
talosctl apply-config --insecure \
  --nodes 192.168.1.100 \
  --file ./_out/controlplane.yaml

# Apply config to the Worker node
talosctl apply-config --insecure \
  --nodes 192.168.1.101 \
  --file ./_out/worker.yaml

Note: The --insecure flag is required only during the initial handoff before node-specific TLS certificates are established.

3. Bootstrap etcd and the Kubernetes Control Plane

After the control plane node reboots, initialize the single-node or multi-node etcd cluster with a single command:

# Execute one-time etcd cluster initialization
talosctl bootstrap --nodes 192.168.1.100

This triggers the PID 1 supervisor to launch etcd, initialize the static pod manifests, and bring up the Kubernetes API server.

4. Retrieve kubeconfig

Generate an admin kubeconfig authenticated against the cluster’s internal PKI:

# Download admin kubeconfig directly via the Talos API
talosctl kubeconfig ./kubeconfig --nodes 192.168.1.100
export KUBECONFIG="$(pwd)/kubeconfig"

# Verify cluster access
kubectl get nodes -o wide

Verification & Monitoring

Because traditional diagnostic utilities like systemctl, top, and journalctl do not exist on the node, all triage happens through talosctl.

1. Inspect Core Services and Runtime Logs

Check the health of core services managed directly by Talos (such as machined, containerd, and kubelet):

# List all internal Talos OS services and their run states
talosctl services --nodes 192.168.1.100

# Stream live logs from the kubelet service
talosctl logs kubelet --nodes 192.168.1.100 --tail 50

2. Built-in Terminal Dashboard

Talos includes a real-time terminal dashboard. It tracks CPU load, memory pressure, network throughput, and container states without requiring third-party monitoring agents:

# Launch the interactive terminal UI
talosctl dashboard --nodes 192.168.1.100

3. Check etcd Health and Hardware State

During control plane troubleshooting, query low-level etcd health and system resources directly:

# Check etcd membership and leader health
talosctl etcd status --nodes 192.168.1.100

# Inspect active network interfaces and storage mounts
talosctl get interfaces --nodes 192.168.1.100
talosctl get disks --nodes 192.168.1.100

4. Atomic, Zero-Downtime OS Upgrades

Upgrading Talos Linux never touches individual packages. Instead, it flips the entire OS image atomically using dual A/B partitions:

# Trigger an atomic OS upgrade on a worker node
talosctl upgrade \
  --nodes 192.168.1.101 \
  --image ghcr.io/siderolabs/installer:v1.8.0 \
  --preserve=true

During this operation, Talos drains running workloads via Kubernetes, writes the new image to the alternate partition, reboots within 15–30 seconds, and runs built-in health checks. If the new kernel fails to initialize, the bootloader reverts to the previous working partition automatically.

Share:
Tags: