Six months ago, our platform team set out to fix a glaring blind spot in our release pipeline. We had no automated check to prove that an image running in our production clusters actually came from our GitHub Actions CI pipeline.
If someone pushed an untracked image directly to our AWS ECR registry, Kubernetes would happily pull and run it. To close that gap across 45 microservices and roughly 250 deployments a week, we rolled out Cosign from the Sigstore project. Here is how container signing works in practice, where it shines, where it bit us, and how to wire it up with admission policies.
Comparing Container Image Signing Approaches
Securing container provenance usually boils down to three options: Docker Content Trust (Notary v1), Notation (Notary v2), and Cosign (Sigstore). Each handles signatures, metadata storage, and key management differently.
| Approach | Signature Storage | Key Management | Adoption Friction |
|---|---|---|---|
| Docker Content Trust (Notary v1) | Separate Notary server and MySQL/Postgres metadata DB | Long-lived client keys (TUF root/targets keys) | High (Requires hosting and scaling separate Notary servers) |
| Notation (Notary v2) | OCI Registry (as OCI Artifact referrers) | X.509 PKI certificates or Cloud KMS | Medium (Built specifically for formal enterprise PKI) |
| Cosign (Sigstore) | OCI Registry (co-located signature manifest or referrer) | Keyless (OIDC + Fulcio + Rekor), static keypair, or KMS | Low (Works on standard OCI registries with zero extra databases) |
Docker Content Trust proved too fragile for fast-moving teams. It required deploying a dedicated Notary server backed by a relational database. When that Notary instance experienced network blips or cache invalidation bugs, production deployments failed across the board.
Cosign solves this by using your existing container registry as the storage backend. When you sign myregistry.com/app:v1.0.0, Cosign uploads a signature manifest directly alongside that image. No external database to maintain. No extra state to sync.
Pros and Cons After 6 Months in Production
Shipping Cosign into an environment handling over 1,200 pods gave us clear data on where the tool excels and where it creates friction.
The Advantages
- Keyless Signing with OIDC: By pairing short-lived OIDC tokens from GitHub Actions with Fulcio (Sigstore’s certificate authority) and Rekor (its transparency log), we completely eliminated long-lived signing keys. Developers never have to generate, rotate, or store private keys.
- Broad Registry Support: Cosign worked out of the box with AWS ECR, Google Artifact Registry, and Harbor without requiring custom plugins or custom API adapters.
- Fast Admission Checks: Verifying signatures at the Kubernetes admission layer with Kyverno added less than 45ms of latency to pod admission webhooks.
- Attestations and SBOMs: Cosign handles more than raw image digests. We use the same CLI workflow to sign and attach CycloneDX Software Bill of Materials (SBOMs) and SLSA build provenance files to our container images.
The Drawbacks
- Metadata Leakage on Public Rekor: Public Sigstore publishes build claims and commit emails to a public ledger. If you run closed-source proprietary software and cannot expose repository URLs or engineer emails, you must either self-host Sigstore’s stack or stick to private Cloud KMS keys.
- Registry Lifecycle Conflicts: Cosign creates tag artifacts formatted as
sha256-[digest].sig. Two weeks after rollout, our ECR lifecycle policy purged untagged base images, which orphaned several signatures and broke staging deployments. We had to rewrite our registry retention rules to explicitly protect*.sigand*.atttags. - Air-Gapped Overhead: Running keyless Sigstore in an isolated VPC or air-gapped data center requires hosting Fulcio, Rekor, Trillian, and an OIDC provider like Dex internally. That setup adds real operational maintenance.
Our Recommended Kubernetes Architecture
For most cloud engineering teams, a hybrid pattern delivers the best balance of security and operational simplicity:
- CI/CD Pipelines: Sign images during build jobs using OIDC keyless authentication in cloud CI (such as GitHub Actions) or Cloud KMS (AWS KMS, Google Cloud KMS, or HashiCorp Vault Transit).
- Cluster Enforcement: Deploy Kyverno or Gatekeeper as a validating admission webhook to block any container lacking a verified cryptographic signature before the pod starts scheduling.
Step-by-Step Implementation Guide
Here is a complete walkthrough for creating keys, signing an image by its immutable digest, and enforcing validation inside Kubernetes.
Step 1: Install the Cosign CLI
Grab the latest binary for your build runner or workstation:
# Download and install Cosign binary (Linux amd64)
LATEST_VERSION=$(curl -s https://api.github.com/repos/sigstore/cosign/releases/latest | grep tag_name | cut -d '"' -f 4)
curl -LO "https://github.com/sigstore/cosign/releases/download/${LATEST_VERSION}/cosign-linux-amd64"
sudo install -o root -g root -m 0755 cosign-linux-amd64 /usr/local/bin/cosign
# Confirm the binary is working
cosign versionStep 2: Generate a Cryptographic Key Pair
If you choose not to use public keyless signing, generate a password-protected keypair:
# Generate cosign.key (private) and cosign.pub (public)
cosign generate-key-pair
# Provide a secure passphrase when promptedStore cosign.key in your CI secret manager (like HashiCorp Vault or AWS Secrets Manager). Keep cosign.pub handy—you will embed it into your Kubernetes policy.
Step 3: Sign the Container Image
Always sign images using their immutable SHA256 digest rather than mutable tags like latest or v1.2.0. Tags can be reassigned; digests cannot.
# Provide the key password in CI pipelines
export COSIGN_PASSWORD="YourProductionSecretPassword"
# Define the exact image digest
IMAGE_DIGEST="registry.example.com/production/auth-service@sha256:7f83b1657ff1fc53b92dc18148a1d65dfc2d4b1fa3d677284addd200126d9069"
# Sign the image
cosign sign --key cosign.key "${IMAGE_DIGEST}"Cosign calculates the signature payload and uploads it directly to your repository as a new manifest layer.
Step 4: Verify the Image Manually
Test the signature from your terminal using the public key:
cosign verify --key cosign.pub "${IMAGE_DIGEST}"If the payload is authentic and the digest matches, Cosign prints the verification claims in JSON format. If the image was altered or signed by an unknown key, Cosign returns a non-zero exit code.
Step 5: Enforce Verification at Admission Time with Kyverno
Manual checks in CI do not stop an engineer from running unverified containers directly via kubectl. We enforce image verification at the cluster boundary using a Kyverno ClusterPolicy.
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: enforce-image-signature
spec:
validationFailureAction: Enforce
background: false
rules:
- name: verify-trusted-signature
match:
any:
- resources:
kinds:
- Pod
verifyImages:
- imageReferences:
- "registry.example.com/production/*"
key: |-
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE7p9r7K9eJ8h9qK3vQ1Z5w8xL
3oR7s+9p4qL8k7m6w5b0x2j3z5v8t1y4u7i0o3p6a9s2d5f8g1h4j7k0l3==
-----END PUBLIC KEY-----Apply the policy to your cluster:
kubectl apply -f verify-policy.yamlTry creating a pod with an unsigned image to test enforcement:
kubectl run test-pod --image=registry.example.com/production/auth-service:untrustedThe Kubernetes API server rejects the request immediately:
Error from server: admission webhook "validate.kyverno.svc" denied the request:
ClusterPolicy enforce-image-signature failed: image signature verification failed for registry.example.com/production/auth-service:untrustedNext Steps
Rolling out signature enforcement does not have to disrupt active development teams. Start by deploying your admission policies with validationFailureAction: Audit. This lets you flag unsigned images in your monitoring dashboards without blocking active deployments. Once your CI pipelines reliably sign 100% of production builds, switch the policy to Enforce mode.
