The Frustrating <pending> Status in On-Premise Kubernetes
You’ve spent hours provisioning a bare-metal Kubernetes cluster. Whether it is a stack of Dell PowerEdges in a rack or three Raspberry Pi 5s on your desk, the setup feels solid. You deploy your first Nginx app, create a LoadBalancer service, and run kubectl get svc. Then you wait. The EXTERNAL-IP column stubbornly shows <pending>. Five minutes pass. Then ten. It never changes.
This is the moment every engineer realizes that bare-metal Kubernetes behaves differently than the cloud. On AWS EKS or Google GKE, the cloud provider automatically spins up a Network Load Balancer (NLB) and bills you for it. In a self-managed environment, that automation doesn’t exist. Kubernetes doesn’t include a network load balancer for bare-metal out of the box. Without an external controller, the cluster has no way to talk to your local router and claim an IP address.
Mastering this networking gap is essential for production-grade on-premise setups. Relying on workarounds like NodePort makes your infrastructure feel fragile and looks unprofessional to end-users who expect standard ports like 80 and 443.
The Root Cause: Missing Cloud Controllers
The problem stems from the missing Cloud Controller Manager (CCM). In a managed environment, the CCM bridges the gap between the Kubernetes API and the provider’s hardware. When you ask for a LoadBalancer, the CCM calls an API, grabs a public IP, and configures the provider’s routers to send traffic to your nodes.
On bare metal, there is no underlying API to call. Kubernetes doesn’t know how to “announce” itself to your network via ARP (Address Resolution Protocol) or BGP (Border Gateway Protocol). Because the cluster can’t tell your home router or office switch that it “owns” a specific IP, the service controller just sits there waiting for a signal that never arrives.
Why Common Workarounds Fail
Before implementing the right fix, let’s look at why other methods usually fall short in professional environments.
1. NodePort: The Clunky Fix
NodePort opens a high-numbered port (30000-32767) on every single node. It works, but it’s ugly. Nobody wants to visit https://myapp.com:31045. Furthermore, if the specific node you are pointing your DNS at goes down, your entire service disappears, even if the rest of the cluster is healthy.
2. ExternalIPs: The Manual Nightmare
You can manually assign an externalIPs field to a service. This tells Kubernetes to accept traffic on that IP if it reaches a node. However, you still have to manually configure your external network to route that traffic. It is a maintenance headache that scales poorly as you add more services.
3. MetalLB: The Industry Standard
MetalLB is a network load balancer designed specifically for bare-metal. It provides a software-based implementation that uses standard routing protocols. It effectively mimics the behavior of a cloud load balancer by “announcing” IPs from a pool you define. It is the most robust way to get real IP addresses for your local services.
Implementing MetalLB: A Step-by-Step Approach
MetalLB integrates natively with the Kubernetes lifecycle. It watches for any service of type LoadBalancer and immediately assigns it an IP from your pre-configured range.
Step 1: Enable Strict ARP Mode
If you use kube-proxy in IPVS mode—standard for clusters with more than 10 nodes for performance reasons—you must enable strict ARP. This allows MetalLB to answer ARP requests correctly. Edit your configuration with this command:
kubectl edit configmap -n kube-system kube-proxySet strictARP to true in the YAML structure:
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: "ipvs"
ipvs:
strictARP: trueStep 2: Install the MetalLB Operator
While Helm is great for versioning, the official manifest is the fastest way to get started. As of version 0.14.3, you can deploy the controller and speaker components with one command:
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.14.3/config/manifests/metallb-native.yamlThe controller handles IP assignments, while the speaker (a DaemonSet) handles the actual networking protocols on each node.
Step 3: Define Your IP Pool
MetalLB needs a dedicated range of IPs. These must be on your local subnet but outside your router’s DHCP range. For example, if your router hands out IPs from .2 to .150, you might use .200 to .250 for Kubernetes.
Create metallb-config.yaml:
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
name: main-static-pool
namespace: metallb-system
spec:
addresses:
- 192.168.1.200-192.168.1.250
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
name: l2-adv
namespace: metallb-system
spec:
ipAddressPools:
- main-static-poolApply it: kubectl apply -f metallb-config.yaml.
Step 4: Verify the Results
Deploy a test service to see it in action. If you have an Nginx deployment ready, run:
kubectl expose deployment nginx --port=80 --type=LoadBalancerCheck the status: kubectl get svc. Within seconds, the <pending> status should disappear, replaced by an IP like 192.168.1.200. You can now hit that IP from any browser on your network.
Layer 2 vs. BGP: Choosing the Right Mode
Your choice depends on your hardware and how much traffic you expect.
Layer 2 Mode (Best for Homelabs)
In Layer 2 mode, one node “wins” an election to host a service IP. It responds to ARP requests for that address. It is simple and works on any basic switch or router (like a TP-Link or Asus). The downside is that all traffic for a specific service hits one node first. It is great for small offices but creates a minor bottleneck for high-bandwidth apps.
BGP Mode (The Production Choice)
BGP is the “pro” move. Each node establishes a peering session with your network router. When a service starts, all nodes tell the router they can handle the traffic. The router then uses Equal-Cost Multi-Path (ECMP) to split traffic across all nodes at the hardware level. This requires a router that supports BGP, such as a Ubiquiti EdgeRouter, MikroTik, or a pfSense box.
Lessons from the Field
After deploying MetalLB across various data centers, I’ve found a few critical rules to follow:
- Exclude IPs from DHCP: If your router gives a laptop the IP
192.168.1.201while MetalLB gives it to an Ingress, you will get intermittent connection drops. These are a nightmare to debug. Always set a manual exclusion in your router settings. - Use Static Node IPs: If a node’s primary IP changes via DHCP, the ARP tables on your switch can get corrupted. Always assign static IPs to your Kubernetes workers.
- IPs are Finite: Don’t give 50 microservices 50 different LoadBalancer IPs. Use MetalLB to give one IP to an Ingress Controller (like Nginx Ingress or Traefik). Then use hostnames (e.g.,
api.example.com) to route traffic internally. - Watch ARP Timeouts: In Layer 2 mode, if a node fails, another takes over the IP. Cheap consumer routers might take 30-60 seconds to update their ARP cache, causing a brief outage during the failover.
The Bottom Line
Moving to bare metal shouldn’t feel like taking a step backward in convenience. MetalLB bridges the gap between your physical hardware and the Kubernetes API, providing a seamless cloud-like experience. Once you move past the <pending> state, your on-prem cluster finally starts behaving like a professional environment. You get the control of bare metal with the automation of the cloud.
