The Breach Nobody Talks About: Your Build Pipeline
A few years ago, I was helping a team audit their Kubernetes deployment after a security incident. Their container images had the right tags, the right registry, the right names — yet something had been tampered with between the CI build and the actual running pod. The culprit wasn’t a zero-day in their app code. It was a compromised build step that slipped a modified binary into a Docker layer.
That incident put supply chain security at the top of my learning list — permanently. Most DevOps teams still treat it as an afterthought.
SolarWinds in 2020, the xz-utils backdoor in 2024, the Codecov compromise — these aren’t exotic edge cases. They’re proof that attackers now target the build process itself, not just the running application.
Root Cause: No Chain of Custody for Artifacts
When you pull an image like myapp:v1.2.3 from a registry, what do you actually know about it? In most pipelines, the honest answer is: not much.
- Was it built from the exact commit tagged
v1.2.3? - Which CI runner built it, and was that runner clean?
- Did any human or script modify the image between build and push?
- Were the base image and dependencies verified before inclusion?
Without answers to these questions, your container registry is basically a box of unlabeled binaries with post-it notes on them.
SLSA — Supply-chain Levels for Software Artifacts — was created by Google and is now maintained by OpenSSF. It addresses exactly this gap. SLSA defines four levels of increasing rigor, each providing stronger guarantees about how software was built and whether it was tampered with.
SLSA Levels at a Glance
- Level 1: Provenance exists — build metadata is generated and available.
- Level 2: Provenance is signed by the build service; history is tamper-evident.
- Level 3: Build runs in an isolated, auditable environment with no operator influence.
- Level 4: Two-party review, hermetic builds — mostly relevant for critical infrastructure.
For most production container workloads, Level 2 is the realistic target and Level 3 is worth reaching for on sensitive services. That’s what I’ll cover here.
Approaches Compared: How Teams Try to Solve This
Option A: Trust Tags and Hope
This is the default. You push v1.2.3, your deployment pulls v1.2.3. No verification happens. It works until it doesn’t — and when it doesn’t, you may not notice for days.
Option B: Manual Image Signing with Docker Content Trust
Docker Content Trust (DCT) uses Notary v1 to sign images. It works, but the operational cost is high: key management is complex, the UX is rough, and most tooling doesn’t integrate cleanly with it. I’ve seen teams enable DCT, fight with it for a week, and quietly turn it off.
Option C: Cosign + SLSA Provenance (the right answer)
Modern pipelines combine Cosign (from Sigstore) for image signing with SLSA provenance attestations generated by your CI system. This gives you:
- Cryptographic proof that the image was built by a specific workflow
- A signed provenance document linking the image to its source commit
- Policy enforcement at deploy time (e.g., with Kyverno or OPA Gatekeeper)
GitHub Actions, GitLab CI, and Google Cloud Build all have native SLSA provenance generators now. The toolchain has matured significantly over the last two years.
Hands-On: SLSA Level 2 for a Container with GitHub Actions
Step 1: Generate SLSA Provenance in Your Build Workflow
GitHub’s slsa-framework/slsa-github-generator handles the heavy lifting. Here’s a minimal workflow:
# .github/workflows/container-slsa.yml
name: Build and Attest Container
on:
push:
tags: ['v*']
jobs:
build:
permissions:
contents: read
packages: write
id-token: write # Required for OIDC signing
runs-on: ubuntu-latest
outputs:
image: ghcr.io/myorg/myapp
digest: ${{ steps.build.outputs.digest }}
steps:
- uses: actions/checkout@v4
- name: Build and push image
id: build
uses: docker/build-push-action@v5
with:
push: true
tags: ghcr.io/myorg/myapp:${{ github.ref_name }}
provenance:
needs: [build]
permissions:
actions: read
id-token: write
packages: write
uses: slsa-framework/slsa-github-generator/.github/workflows/[email protected]
with:
image: ${{ needs.build.outputs.image }}
digest: ${{ needs.build.outputs.digest }}
registry-username: ${{ github.actor }}
secrets:
registry-password: ${{ secrets.GITHUB_TOKEN }}Once it runs, the attestation lands in your registry as an OCI artifact alongside the image — no separate database or storage needed. docker/build-push-action outputs the image digest directly as steps.build.outputs.digest, so no extra export step is required.
Step 2: Verify the Image Before Deployment
Install the Cosign CLI on your workstation or in your CD pipeline:
# Install cosign
curl -LO https://github.com/sigstore/cosign/releases/latest/download/cosign-linux-amd64
chmod +x cosign-linux-amd64
sudo mv cosign-linux-amd64 /usr/local/bin/cosign
# Install slsa-verifier
curl -LO https://github.com/slsa-framework/slsa-verifier/releases/latest/download/slsa-verifier-linux-amd64
chmod +x slsa-verifier-linux-amd64
sudo mv slsa-verifier-linux-amd64 /usr/local/bin/slsa-verifierThen verify provenance before pulling or deploying:
# Verify SLSA provenance — confirms the image was built from your repo
slsa-verifier verify-image \
ghcr.io/myorg/myapp@sha256:abc123... \
--source-uri github.com/myorg/myrepo \
--source-tag v1.2.3
# Expected output:
# PASSED: Verified SLSA provenance
# slsa_level: 3Always verify by digest, not by tag. Tags are mutable — a digest is not.
Step 3: Also Sign the Image Itself with Cosign
SLSA provenance covers the build process. Cosign signing covers the image artifact itself. Use both:
# Keyless signing (uses GitHub OIDC — no key management needed)
cosign sign --yes ghcr.io/myorg/myapp@sha256:abc123...
# Verification
cosign verify \
--certificate-identity-regexp 'https://github.com/myorg/myrepo/.github/workflows/' \
--certificate-oidc-issuer https://token.actions.githubusercontent.com \
ghcr.io/myorg/myapp@sha256:abc123...Keyless signing (via Sigstore’s Fulcio CA and Rekor transparency log) removes the private key management headache entirely. The identity comes from your GitHub Actions OIDC token — short-lived cryptographic proof bound to the actual workflow run, not a long-lived secret that someone might accidentally leak or commit to a repo.
Enforcing Verification at Deploy Time
Signing images is worthless if nothing checks the signatures before running them. In Kubernetes, a policy admission controller is your enforcement point. Here’s a Kyverno policy that blocks unsigned images:
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: require-image-signature
spec:
validationFailureAction: Enforce
rules:
- name: verify-cosign-signature
match:
resources:
kinds: [Pod]
verifyImages:
- imageReferences:
- "ghcr.io/myorg/*"
attestors:
- entries:
- keyless:
subject: "https://github.com/myorg/myrepo/.github/workflows/container-slsa.yml@refs/tags/*"
issuer: "https://token.actions.githubusercontent.com"
rekor:
url: https://rekor.sigstore.devAny pod attempting to run an unsigned or tampered image from your registry gets rejected by the API server before it ever starts.
What SLSA Level 3 Adds
Using generator_container_slsa3.yml already achieves SLSA Level 3 provenance quality. The generator runs in an isolated environment that your main build job cannot touch or modify. The key properties at L3:
- Provenance is generated in an environment the build job cannot influence.
- The build definition (your Dockerfile + workflow) is captured in the provenance.
- Signing happens via a hardware-backed identity (GitHub OIDC), not a developer-managed key.
Practical Tips from Real Deployments
- Pin your base images by digest too.
FROM ubuntu:22.04is a moving target. UseFROM ubuntu@sha256:abc...so your provenance actually means something when someone audits it later. - Store provenance alongside your artifacts. OCI registries that support OCI 1.1 — GHCR, ECR, GCR — store attestations as referrers. No separate database needed.
- Wire verification into your CD pipeline, not just developers’ laptops. A manual step that someone can skip isn’t a control.
- Log all verification events. When signature verification fails, you want that in your SIEM. Cosign and slsa-verifier both return non-zero exit codes on failure, so wrapping them in pipeline steps is straightforward.
- Don’t skip SBOM generation. SLSA provenance tells you how the image was built; an SBOM tells you what it contains. Use
syftordocker buildx‘s built-in SBOM support alongside SLSA attestations.
Common Mistakes to Avoid
Teams most often stumble here: implementing signing and skipping verification entirely. Signing is the easy part. The value only shows up when you’re actually checking signatures at the enforcement boundary — the admission controller, the CD gate, the policy engine.
Mutable image tags in verification policies are the second trap. A policy that verifies myapp:latest can be satisfied by any signed image pushed as latest — that’s no protection against tag hijacking. Reference digests in production deployments, always.
Finally, don’t treat SLSA as a compliance checkbox. Level 1 provenance with no enforcement is nearly worthless. The goal is to make it verifiably detectable when a tampered artifact tries to reach production — not to check a box on a security questionnaire.
Where to Go From Here
Once you’ve got SLSA Level 2–3 wired up for containers, there are a few worthwhile extensions:
- Extend the same pipeline to non-container artifacts — Helm charts, binaries, Lambda ZIPs — using
slsa-github-generator‘s generic artifact workflows. - Set up Rekor log monitoring to alert on unexpected signing events for your identity.
- Check your dependencies’ own SLSA levels. Tools like
deps.devnow surface SLSA provenance data for open source packages.
Supply chain security has no finish line. Every layer you harden is one fewer entry point for the next SolarWinds. Wiring SLSA provenance into your container pipeline is one of the most concrete, verifiable improvements you can ship right now.
