Why Your Bash Scripts Are Failing You
Deploying a YAML file is the easy part. Keeping a complex application healthy in production is where things get messy. I often see teams leaning on a patchwork of Bash scripts, Jenkins pipelines, or manual kubectl hacks to handle operational logic. These scripts usually tackle tasks like database schema migrations, shard rebalancing, or certificate rotation.
The real headache is that scripts are “imperative.” They run a sequence of steps and then stop. If a network blip kills your script at step 4 of 10, you are left with a half-baked deployment and a 2 AM incident call. Scripts have no memory. If the environment drifts five minutes after the script finishes, nothing is there to fix it. This creates “operational debt” that forces engineers to babysit automation instead of shipping features.
The Intelligence Gap in Standard Controllers
Kubernetes relies on a Control Loop. Built-in controllers for Deployments or StatefulSets work 24/7 to ensure your current state matches your desired state. However, these controllers are generic. They know a Pod should be “Running,” but they don’t know that your specific app needs a 30-second warm-up period or a specific API registration call before it can take traffic.
Standard resources lack the “domain intelligence” of your specific stack. When we bridge this gap with external scripts, we lose the declarative magic of Kubernetes. We need a way to teach the cluster how to manage our application as if it were a first-class, native resource.
The Solution: Kubernetes Operators
An Operator is essentially human operational knowledge packaged into software. It combines Custom Resource Definitions (CRDs) with a Custom Controller. Instead of firing off a script, you define your “Desired State” in a YAML file. The Operator then works continuously to make that state a reality.
To build these efficiently, we use the controller-runtime library. This is the same engine powering the Operator SDK and Kubebuilder. It handles the boring plumbing—like API communication, resource caching, and event queuing—so you can focus entirely on your business logic. In my experience, switching to this model can reduce recovery time (MTTR) from 15 minutes of manual intervention to under 500ms of automated reconciliation.
Hands-on: Building Your First Operator
Let’s build a basic Operator for a custom resource called AppService. This Operator will ensure that every AppService automatically maintains its own Deployment and ConfigMap.
1. Initialize the Project
You will need Go and the Operator SDK. Start by creating a fresh directory and initializing the scaffolding:
mkdir app-operator
cd app-operator
operator-sdk init --domain itfromzero.com --repo github.com/itfromzero/app-operator2. Create the API
Now, we define our Custom Resource. This tells the Kubernetes API server about our new object type.
operator-sdk create api --group apps --version v1alpha1 --kind AppService --resource --controller3. Define the Desired State
Open api/v1alpha1/appservice_types.go. We need to tell Kubernetes what fields our resource expects, such as the number of replicas and a configuration message.
type AppServiceSpec struct {
Size int32 `json:"size"`
Message string `json:"message"`
}
type AppServiceStatus struct {
Nodes []string `json:"nodes"`
}After updating these structs, run make generate and make manifests. This updates the auto-generated code and the YAML files that define your CRD.
4. Implementing the Reconcile Loop
The Reconcile function in internal/controller/appservice_controller.go is where the magic happens. It triggers whenever an AppService changes. Think of the logic as: Observe -> Analyze -> React.
func (r *AppServiceReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
log := log.FromContext(ctx)
// 1. Fetch the AppService instance
appService := &appsv1alpha1.AppService{}
err := r.Get(ctx, req.NamespacedName, appService)
if err != nil {
return ctrl.Result{}, client.IgnoreNotFound(err)
}
// 2. Check if the Deployment exists; create it if missing
found := &appsv1.Deployment{}
err = r.Get(ctx, types.NamespacedName{Name: appService.Name, Namespace: appService.Namespace}, found)
if err != nil && errors.IsNotFound(err) {
dep := r.deploymentForAppService(appService)
log.Info("Creating a new Deployment", "Namespace", dep.Namespace, "Name", dep.Name)
err = r.Create(ctx, dep)
if err != nil {
return ctrl.Result{}, err
}
return ctrl.Result{Requeue: true}, nil
}
// 3. Sync the deployment size with the spec
size := appService.Spec.Size
if *found.Spec.Replicas != size {
found.Spec.Replicas = &size
err = r.Update(ctx, found)
if err != nil {
return ctrl.Result{}, err
}
}
return ctrl.Result{}, nil
}5. Deploy and Test
Use Kind (Kubernetes in Docker) for local testing. Install the CRDs and run your controller with these commands:
make install
make runIn a separate terminal, apply a sample resource:
apiVersion: apps.itfromzero.com/v1alpha1
kind: AppService
metadata:
name: example-app
spec:
size: 3
message: "Hello from Operator"Once you run kubectl apply, watch the logs. The controller will immediately detect the new resource and spin up your pods.
The Self-Healing Advantage
Why go through all this trouble? The answer is Self-Healing. If a junior engineer accidentally deletes your Deployment, the Reconcile loop catches it instantly. It observes the missing resource, analyzes the discrepancy, and recreates it in milliseconds. A cron-based Bash script might eventually fix this, but an Operator provides real-time resilience.
Go also gives you access to envtest and robust unit testing frameworks. You can simulate complex failures, like API timeouts or network partitions, ensuring your logic is production-ready before it ever touches a cluster.
Conclusion
Moving from manual scripts to custom Operators is a major milestone for any DevOps team. It transforms “tribal knowledge”—the expertise locked in a senior engineer’s head—into reliable, versioned code. While learning Go and the Kubernetes API takes effort, the stability and scalability you gain are worth every hour of study.
