Stop Managing Cloud Resources Manually: A Guide to Cloud Custodian

The Manual Audit Trap: Why Cloud Sprawl Kills Productivity

Cloud infrastructure scales faster than your team’s ability to track it. One week you’re managing ten EC2 instances; the next, your organization has exploded into hundreds of accounts across AWS, Azure, and GCP.

I’ve watched senior DevOps engineers lose their entire Friday hunting for unencrypted S3 buckets or deleting “zombie” disks that were forgotten months ago. A single orphaned 1TB gp3 volume in AWS costs roughly $80 per month for doing absolutely nothing. Multiply that by fifty volumes across ten regions, and you’re burning $4,000 monthly on digital trash.

The friction usually stems from a lack of centralized, automated enforcement. Relying on engineers to memorize a 50-page “Best Practices” PDF is a recipe for failure. Human error is inevitable. To maintain control, you need to transform those guidelines into executable code. This is where Cloud Custodian (c7n) becomes your most valuable DevOps tool.

Quick Start: Deploy Your First Policy in 5 Minutes

Cloud Custodian is an open-source rules engine that uses YAML to define infrastructure policies. It is remarkably lightweight. You don’t need to manage a massive server cluster to run it; a local laptop or a basic CI/CD runner is more than enough.

1. Installation

Since Cloud Custodian is Python-based, a virtual environment is the best way to keep your workspace clean.

python3 -m venv custodian-venv
source custodian-venv/bin/activate
pip install c7n

2. Write a Security Policy

Create a file named encrypt-buckets.yml. This policy scans for any S3 bucket lacking AES-256 or KMS encryption and applies a warning tag. It’s a non-destructive way to audit your security posture.

policies:
  - name: s3-unencrypted-report
    resource: s3
    filters:
      - type: no-encryption
    actions:
      - type: tag
        key: SecurityStatus
        value: NonCompliant

3. Execute the Audit

Run the policy against your AWS account. Ensure your credentials are exported in your terminal session before starting.

custodian run --output-dir=. encrypt-buckets.yml

Custodian will generate a resources.json file. This file lists every bucket that failed the check. No more clicking through the AWS Console for hours.

The Three Pillars: How Policies Are Built

Every Cloud Custodian policy follows a logical, three-part structure. Think of it as an “If-This-Then-That” for your cloud infrastructure.

• Resource: The target asset (e.g., aws.ec2, azure.vm, or gcp.instance).
• Filter: The specific criteria for selection. You might target “instances older than 60 days” or “disks with no attached owner.”
• Action: The response. You can stop, delete, tag, or even send a Slack notification to the resource creator.

In production environments, I use this logic to handle orphaned EBS volumes every Sunday night. It’s a simple way to keep the “cloud graveyard” from draining the quarterly budget.

Multi-Cloud Governance in Practice

Cloud Custodian abstracts provider-specific APIs into a unified syntax. This means you don’t have to learn three different CLI tools to manage a multi-cloud environment.

AWS: Automating Cost Savings

Development environments often stay running over the weekend, wasting 60+ hours of compute time. This policy targets instances tagged Env: Dev and shuts them down at 6:00 PM.

policies:
  - name: ec2-nightly-stop
    resource: ec2
    filters:
      - type: value
        key: "tag:Env"
        value: Dev
      - type: schedule
        schedule: "stop"
        default_tz: "utc"
    actions:
      - stop

Azure: Cleaning Up Managed Disks

Azure continues to bill you for managed disks even after the parent VM is deleted. This policy finds and removes those unattached disks automatically.

policies:
  - name: azure-cleanup-unattached-disks
    resource: azure.disk
    filters:
      - type: value
        key: properties.diskState
        op: eq
        value: Unattached
    actions:
      - type: delete

Production Strategies for Peace of Mind

Running a tool that can delete infrastructure is naturally intimidating. To avoid accidental outages, I follow a strict deployment workflow.

The “Dry Run” Safety Net

Never run a new policy in “live” mode first. Use the --dryrun flag. It simulates the execution and tells you exactly which resources would have been affected without actually changing anything.

custodian run --dryrun --output-dir=out policy.yml

The “Notify and Wait” Grace Period

Sudden deletions frustrate developers. Instead of immediate action, create a two-step policy. The first policy tags a non-compliant resource and sends a Slack alert. If the resource remains non-compliant after a 72-hour grace period, a second policy performs the deletion.

CI/CD Integration

Long-term governance shouldn’t happen on your laptop. Move your YAML policies into a Git repository. Use GitHub Actions or GitLab CI to run these policies on a schedule—perhaps every hour for security checks and once a week for cost cleanup.

Final Thoughts

Cloud governance isn’t about playing “infrastructure police.” It’s about building a guardrail system that allows developers to move fast without leaving the front door open. By treating policies as code, you stop reacting to billing surprises and start managing your cloud with intent. Cloud Custodian turns a chaotic environment into a predictable, automated ecosystem.