The Problem That Bites You in Production
You’ve written your Kubernetes deployment YAML, pushed it, and everything looks fine in staging. Then, three weeks later, a pod gets OOMKilled in production because someone forgot to set resource limits. Or worse — your cluster gets partially compromised because a container is running as root with a permissive security context that nobody noticed.
This isn’t a hypothetical scenario. In my experience working with Kubernetes clusters across multiple environments, manifest quality issues are one of the most consistent sources of production incidents. The root cause is almost always the same: YAML files get written quickly, reviewed superficially in pull requests, and nobody catches the missing readinessProbe or the allowPrivilegeEscalation: true lurking in the spec.
The fix is straightforward: static analysis for your Kubernetes manifests, baked into your CI/CD pipeline so issues get caught before they ever reach a cluster. In my real-world experience, this is one of the essential skills to master when maintaining Kubernetes infrastructure at any serious scale.
Two Tools Worth Knowing
There are two tools I reach for when setting up manifest quality gates: Kube-score and Kube-linter. They complement each other well — Kube-score focuses on general best practices and gives each resource a score, while Kube-linter leans heavier into security-specific checks. Using both gives solid coverage without much overlap.
Kube-score
Kube-score reads your manifest files and evaluates each resource against a set of best practice checks. It flags issues like missing resource requests/limits, missing health probes, containers running as root, and non-pinned image tags. Each finding is categorized by severity: CRITICAL, WARNING, or OK. The scoring model makes it easy to track improvement over time.
Kube-linter
Kube-linter, originally from StackRox (now part of Red Hat), runs 30+ built-in checks with a strong emphasis on security posture. It catches privileged containers, missing network policies, overly permissive RBAC, and similar misconfigurations that kube-score might treat more leniently. The two tools have different check philosophies, which is exactly why running both is worth the small extra CI time.
Hands-on: Installing and Running Both Tools
Installing Kube-score
Grab the binary from GitHub releases:
# Linux (x86_64)
curl -Lo kube-score https://github.com/zegl/kube-score/releases/latest/download/kube-score_linux_amd64
chmod +x kube-score
sudo mv kube-score /usr/local/bin/
# macOS via Homebrew
brew install kube-score
# Verify installation
kube-score versionInstalling Kube-linter
# Linux
curl -Lo kube-linter https://github.com/stackrox/kube-linter/releases/latest/download/kube-linter-linux
chmod +x kube-linter
sudo mv kube-linter /usr/local/bin/
# macOS via Homebrew
brew install kube-linter
# Verify installation
kube-linter versionRunning Your First Check
Take a typical deployment manifest that looks passable on a quick review:
# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
spec:
replicas: 1
selector:
matchLabels:
app: my-app
template:
metadata:
labels:
app: my-app
spec:
containers:
- name: my-app
image: my-app:latestRun kube-score against it:
kube-score score deployment.yamlThe output surfaces several real problems:
apps/v1/Deployment my-app 💥 CRITICAL
[CRITICAL] Container Security Context
Container 'my-app' does not have a security context set
[CRITICAL] Container Resources
Container 'my-app' does not have resource requests set
Container 'my-app' does not have resource limits set
[WARNING] Container Liveness Probe
Container 'my-app' does not have a liveness probe set
[WARNING] Container Readiness Probe
Container 'my-app' does not have a readiness probe set
[WARNING] Deployment has a single replica — not resilient to node failuresNow run kube-linter on the same file:
kube-linter lint deployment.yamldeployment.yaml: container "my-app" does not have a read-only root file system (check: no-read-only-root-fs)
deployment.yaml: container "my-app" has no resource limits (check: resource-limits)
deployment.yaml: container "my-app" is not set to runAsNonRoot (check: run-as-non-root)
deployment.yaml: container "my-app" has an image using a mutable tag (check: no-latest-image)From a single 18-line YAML, both tools surfaced genuine production risks. That’s the value of automated linting.
A Corrected Manifest
Here’s the same deployment after addressing the critical findings:
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
spec:
replicas: 2
selector:
matchLabels:
app: my-app
template:
metadata:
labels:
app: my-app
spec:
securityContext:
runAsNonRoot: true
runAsUser: 1000
containers:
- name: my-app
image: my-app:1.2.3 # pinned tag — never :latest
resources:
requests:
cpu: "100m"
memory: "128Mi"
limits:
cpu: "500m"
memory: "256Mi"
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop:
- ALL
livenessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 10
periodSeconds: 15
readinessProbe:
httpGet:
path: /ready
port: 8080
initialDelaySeconds: 5
periodSeconds: 10Wiring Both Tools into CI/CD with GitHub Actions
Running these tools locally is useful, but the real value comes from making them required gates in your pipeline. Every pull request that touches Kubernetes manifests should run both checks automatically — no exceptions, no bypass.
# .github/workflows/k8s-lint.yml
name: Kubernetes Manifest Checks
on:
pull_request:
paths:
- 'k8s/**'
- 'manifests/**'
- '**/*.yaml'
- '**/*.yml'
jobs:
kube-score:
name: Kube-score Analysis
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Install kube-score
run: |
curl -Lo kube-score https://github.com/zegl/kube-score/releases/latest/download/kube-score_linux_amd64
chmod +x kube-score && sudo mv kube-score /usr/local/bin/
- name: Run kube-score
run: |
find k8s/ -name "*.yaml" | \
xargs kube-score score --exit-one-on-warning
kube-linter:
name: Kube-linter Security Checks
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Run kube-linter
uses: stackrox/kube-linter-action@v1
with:
directory: k8s/
config: .kube-linter.yamlThe --exit-one-on-warning flag makes kube-score fail the pipeline on warnings too, not just critical issues. I usually start without this flag when onboarding an existing project (to avoid an overwhelming number of failures on day one), then add it once the critical findings are cleared.
Customizing Kube-linter Rules
Not every check applies to every environment. Kube-linter supports a config file for enabling or disabling specific checks:
# .kube-linter.yaml
checks:
addAllBuiltIn: true
exclude:
- "minimum-three-replicas" # staging runs 2 replicas intentionally
- "required-annotation-email" # not using this annotation conventionList all available checks with:
kube-linter checks listLinting Helm-rendered Output
If your project uses Helm, lint the rendered manifests — not the raw templates. Template-level linting misses values-driven configuration problems:
# Pipe helm template output directly to kube-score
helm template my-release ./chart -f values.prod.yaml | kube-score score -
# Same for kube-linter
helm template my-release ./chart -f values.prod.yaml > /tmp/rendered.yaml
kube-linter lint /tmp/rendered.yamlJSON Output for CI Dashboards
For teams that want to track findings over time or integrate with reporting tools, kube-score supports JSON output:
# Show only CRITICAL findings from JSON output
kube-score score deployment.yaml --output-format json | \
jq '[.[] | .checks[] | select(.grade == "CRITICAL")]'Practical Tips from Working with Real Clusters
- Triage incrementally. Adding these tools to a mature project usually surfaces dozens of findings. Fix CRITICALs first, then work through WARNINGs over subsequent sprints rather than trying to clean everything in one PR.
- Resource limits protect the whole cluster. A single pod without limits can starve neighboring workloads through noisy-neighbor effects. This is a cluster stability issue, not just a pod configuration detail.
- Pinned image tags are non-negotiable. Both tools flag
:latestand similar mutable tags. Unpinned images make deployments non-reproducible and can introduce unexpected changes during rolling restarts. - Run checks in pre-commit hooks too. Use pre-commit to run kube-score locally before commits reach CI — faster feedback loop, fewer failed pipelines.
Where to Go From Here
Once kube-score and kube-linter are running in your pipeline, you’ve eliminated the most common class of Kubernetes manifest problems at the source. The natural next step is pairing static CI checks with runtime admission control — tools like Kyverno or OPA/Gatekeeper enforce the same rules at the cluster level, so even manifests applied directly via kubectl get validated before pods are scheduled.
But the CI integration alone is already a significant improvement. Get these checks running on pull requests, fix the CRITICALs systematically, and within a few sprints your manifest quality will be in a genuinely different place than where you started.
