Posted inSecurity

Kubernetes Security Policies with Kyverno: Enforcing Admission Control in Production

Security tutorial - IT technology blog
Security tutorial - IT technology blog

The Problem: The Hidden Chaos of Kubernetes Misconfigurations

Across eight engineering squads and three production clusters running over 450 pods, our deployment velocity was sky-high. Security guardrails, however, were virtually nonexistent. During a single routine audit, we uncovered 14 workloads running as root, two troubleshooting pods running with full privileged: true host access, and dozens of deployments completely omitting CPU and memory limits.

Leaving admission control unconfigured in Kubernetes is the modern equivalent of exposing a root SSH login to the open internet. Relying on pull request reviews or manual checklists fails the moment teams ship 30 deployments a day. Bad configs inevitably slip through. We needed automated guardrails directly at the API server boundary to reject risky configurations before they ever reached worker nodes.

We initially tested Open Policy Agent (OPA) Gatekeeper. While capable, writing and maintaining policies in Rego created an unnecessary learning curve for our developers and platform engineers. That friction led us straight to Kyverno.

Core Concepts: Why Kyverno Fits Native Kubernetes Workflows

Kyverno runs as a dynamic admission controller inside your cluster. It registers with the Kubernetes API server via ValidatingAdmissionWebhook and MutatingAdmissionWebhook endpoints to inspect resource manifests during every create, update, and delete request.

Unlike engines that require domain-specific programming languages, Kyverno policies are standard Kubernetes Custom Resource Definitions (CRDs). If your team knows how to write a Pod spec in YAML, they already know how to write a Kyverno policy.

The Four Policy Modes

  • Validate: Evaluates incoming manifests against rules. It blocks non-compliant requests in Enforce mode or permits them while logging warnings in Audit mode.
  • Mutate: Modifies resources on the fly before persisting them. Typical use cases include injecting default security contexts, applying cost-center labels, or adding sidecars.
  • Generate: Creates supporting resources automatically upon trigger events, such as provisioning default NetworkPolicies and RoleBindings whenever someone creates a Namespace.
  • Verify Images: Checks container signatures against Cosign/Sigstore keys and validates software bill of materials (SBOM) attestations at admission time.

Hands-on Practice: Deploying Kyverno and Securing Your Cluster

Here is the exact setup and policy suite we deployed to eliminate privileged containers and enforce non-root execution across our clusters.

Step 1: Installing Kyverno via Helm

Install Kyverno into its dedicated namespace using the official Helm chart. For production environments, run three replicas to maintain high availability during cluster upgrades:

# Add and update the Kyverno Helm repository
helm repo add kyverno https://kyverno.github.io/kyverno/
helm repo update

# Deploy Kyverno with HA replicas
helm install kyverno kyverno/kyverno \
  --namespace kyverno \
  --create-namespace \
  --set replicaCount=3

Confirm that the controller pods are healthy and the webhooks are registered:

kubectl get pods -n kyverno
kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations

Step 2: Blocking Privileged Containers (Validation Rule)

Setting privileged: true disables container isolation mechanisms and grants near-total access to the host kernel. This ClusterPolicy blocks privileged containers cluster-wide:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-privileged-containers
  annotations:
    policies.kyverno.io/title: Disallow Privileged Containers
    policies.kyverno.io/category: Pod Security Standards
    policies.kyverno.io/severity: high
    policies.kyverno.io/description: >-
      Privileged containers share root capabilities with the host operating system.
      This policy ensures no container can run in privileged mode.
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: validate-privileged
      match:
        any:
          - resources:
              kinds:
                - Pod
      validate:
        message: "Privileged mode is strictly forbidden. Set securityContext.privileged to false."
        pattern:
          spec:
            containers:
              - =(securityContext):
                  =(privileged): "false"

Apply the policy to your cluster:

kubectl apply -f disallow-privileged-containers.yaml

Step 3: Mutating Resources to Enforce Non-Root Execution

Instead of rejecting every deployment where a developer forgot to declare a security context, use a mutating policy. It automatically injects runAsNonRoot: true whenever the field is missing:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: mutate-non-root-execution
spec:
  rules:
    - name: set-run-as-non-root
      match:
        any:
          - resources:
              kinds:
                - Pod
      mutate:
        patchStrategicMerge:
          spec:
            securityContext:
              runAsNonRoot: true

Step 4: Testing Policy Enforcement in Real Time

Verify enforcement by attempting to deploy a test pod configured with host privileges:

apiVersion: v1
kind: Pod
metadata:
  name: rogue-priv-pod
spec:
  containers:
    - name: alpine-test
      image: alpine:latest
      command: ["sleep", "3600"]
      securityContext:
        privileged: true

Submit the manifest using kubectl:

kubectl apply -f rogue-priv-pod.yaml

The API server immediately blocks admission and returns your custom error message:

Error from server: error when creating "rogue-priv-pod.yaml": admission webhook "validate.kyverno.svc" denied the request:

policy Pod/default/rogue-priv-pod for resource violation:

disallow-privileged-containers:
  validate-privileged: Privileged mode is strictly forbidden. Set securityContext.privileged to false.

Step 5: Auditing Existing Workloads with Policy Reports

In addition to intercepting live API calls, Kyverno runs periodic background scans against running workloads. You can query cluster compliance directly using PolicyReport custom resources:

kubectl get clusterpolicyreports -o wide
kubectl describe policyreport -n default

This report highlights non-compliant workloads deployed before policy activation, allowing teams to remediate drift gradually without causing sudden outages.

Reflections After Six Months in Production

Over the past six months, Kyverno processed more than 200,000 admission requests across our clusters with an average webhook latency under 4 milliseconds. During that period, it caught and rejected 47 invalid or privileged configurations before they reached runtime.

Because policies use standard Kubernetes YAML, developers do not need to learn a custom DSL to understand admission rejections. They read the error message, update their deployment manifest, and move on. If you need robust cluster governance without operational overhead, Kyverno delivers exactly that.

Share:
Tags: