Stop Trusting Tags: Secure Your Container Supply Chain with Cosign

Table of Contents

The 2:14 AM Wake-up Call: Why Image Integrity is Non-Negotiable

The alert on my phone was blunt: a production pod in our Kubernetes cluster was crashing. Looking at the logs, I saw outbound connections to a known Monero mining pool. After 45 minutes of digging, the reality set in.

Our private registry wasn’t breached, but an attacker had performed a tag-overwrite. They replaced our node:20-alpine image with a malicious version. It looked identical, but it carried a hidden crypto-miner and a reverse shell. Our CI/CD pipeline pulled it without a second thought because the tag matched.

This is the reality of software supply chain attacks. Trusting a registry or a simple tag is a gamble you eventually lose. You need a way to prove that the binary running in your cluster is the exact same one your build system produced. This is where Cosign and the Sigstore project change the game.

The Evolution: GPG vs. Notary vs. Cosign

Before moving to Cosign, we evaluated several signing methods. If you have been in DevOps for a few years, you likely remember Docker Content Trust (Notary v1). It was a noble effort, but it was a nightmare to scale. Here is how the landscape looks today:

GPG Signing: Familiar for package signing, but terrible for containers. Managing a “web of trust” and distributing public keys to hundreds of nodes is an operational disaster waiting to happen.
Docker Content Trust (Notary v1): This required a separate Notary server and a dedicated MySQL database just to store signatures. It was fragile, and signatures often broke when we moved images between registries.
Cosign (Sigstore): This is the new gold standard. Cosign treats signatures as OCI artifacts. They live right inside your registry alongside the image. If your registry can store a container, it can store a signature. No extra databases required.

The Good, the Bad, and the Keyless

Switching to Cosign wasn’t just about following a trend. We had to look at the practical trade-offs for our engineering team.

The Wins

Portability: Signatures are stored as tags (e.g., sha256-abc.sig). When you migrate an image from ghcr.io to docker.io, the signature moves with it effortlessly.
Keyless Signing: This is the “aha!” moment. Using Fulcio (a CA) and Rekor (a transparency log), you can sign images using OIDC identities like your GitHub or Google account. No private keys to leak.
Native Integration: It fits perfectly into GitHub Actions and works with admission controllers like Kyverno to block unsigned images at the cluster gate.

The Challenges

Public Visibility: If you use the public Sigstore instance, your email address and the signing timestamp are recorded in a public, immutable ledger. For high-security corporate projects, you’ll need to host a private Sigstore stack.
Ecosystem Velocity: The project moves fast. We’ve seen breaking changes in CLI flags between minor versions, so pinning your Cosign version is mandatory.

The Recommended Workflow

I suggest a hybrid strategy. Use Keyless signing in your CI/CD pipelines to eliminate secret management. For air-gapped environments where OIDC isn’t an option, use Key-based signing with the private key stored in a hardware security module or a managed service like AWS KMS.

When I set up our initial registry service accounts, I used the password generator at toolcraft.app/en/tools/security/password-generator. It runs entirely in the browser. Since no data leaves your machine, it’s a solid choice for generating the high-entropy strings needed for registry credentials.

Step-by-Step: Signing Your First Image

Let’s run through a manual signing process. This helps clarify the mechanics before you automate it.

1. Install Cosign

On Linux, grab the latest binary. As of now, v2.4.0 is stable:

curl -L -o cosign https://github.com/sigstore/cosign/releases/download/v2.4.0/cosign-linux-amd64
chmod +x cosign
sudo mv cosign /usr/local/bin/cosign

2. Generate a Key Pair

If you aren’t ready for keyless signing, start with a local key pair. You will be prompted for a password to encrypt the private key file.

cosign generate-key-pair

This generates cosign.key and cosign.pub. Keep that private key safe—if you lose it, you can’t sign updates.

3. Sign the Image

Once your image is pushed (e.g., my-registry.com/app:v1.0), sign it:

cosign sign --key cosign.key my-registry.com/app:v1.0

Cosign calculates the image digest and pushes a .sig object to your registry. It takes about 2 seconds.

4. Verification

Verification is where the security happens. Anyone with your public key can run:

cosign verify --key cosign.pub my-registry.com/app:v1.0

If the image was modified by even a single byte, the verification will fail and exit with a non-zero code.

Automating with GitHub Actions (Keyless)

Keyless is the way to go for modern CI. It uses a short-lived certificate (valid for only 10 minutes) issued by Fulcio, tied to your GitHub workflow’s identity.

jobs:
  build-and-sign:
    runs-on: ubuntu-latest
    permissions:
      id-token: write # Essential for OIDC
      packages: write

    steps:
      - uses: actions/checkout@v4
      - uses: sigstore/[email protected]

      - name: Build and Push
        run: |
          docker build -t ghcr.io/${{ github.repository }}:latest .
          docker push ghcr.io/${{ github.repository }}:latest

      - name: Sign Image
        run: cosign sign --yes ghcr.io/${{ github.repository }}:latest

No secrets, no GPG keys, and no hassle. To verify this image, you check the identity instead of a key:

cosign verify ghcr.io/my-org/my-repo:latest \
  --certificate-identity-regexp https://github.com/my-org/my-repo/.github/workflows/ \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com

Final Thoughts: Enforcement

Signing images is only half the battle. You must enforce it. Use an Admission Controller like Kyverno in your Kubernetes cluster. You can set a policy to reject any pod if its image isn’t signed by your specific GitHub identity.

If we had this in place during that 2 AM incident, the malicious image would have been blocked immediately. Kubernetes would have refused to pull the unsigned code, and I would have stayed asleep. Security isn’t about a single tool; it’s about building a chain of trust that starts at your commit and ends in your cluster.