Context & Why
At 2:15 AM on a Thursday, PagerDuty woke me up for a production hotfix stuck in queue. Thirty integration test suites had jammed our GitHub Actions pipelines.
We hit GitHub’s 20-concurrency hosted cap, while our four static c5.2xlarge EC2 runners pegged at 100% CPU. Meanwhile, 64 idle CPU cores sat untouched in our staging EKS cluster right next door. The breakdown was clear: static runner fleets burn thousands of dollars every weekend when nobody commits code, yet they choke during release rushes.
Standard GitHub-hosted runners are convenient, but team bills multiply fast. At $0.008 per minute for standard Linux compute, running 50,000 build minutes a month quickly burns through budgets.
More critically, hosted runners cannot reach internal databases, private registries, or pre-warmed container caches inside a VPC. Static VMs bypass those network walls, but paying for them 24/7 wastes cash. The better path decouples build workloads from persistent VMs and delegates lifecycle duties directly to a Kubernetes operator.
Actions Runner Controller (ARC) is an official Kubernetes operator backed by GitHub. It tracks workflow queues via an HTTPS long-poll connection and provisions ephemeral runner pods inside your cluster on demand. Pods terminate the second a job finishes. No dirty state, no cross-job disk pollution, and zero compute costs when developers log off for the day.
Installation
To deploy ARC’s autoscaling architecture, you need a Kubernetes cluster (v1.25+) and Helm 3.8+. ARC splits duties between two Helm charts: a controller manager that manages custom resources, and scale sets that represent individual worker pools.
1. Configure GitHub Authentication
ARC authenticates to your organization or repository using a GitHub App. This avoids the strict 5,000 requests-per-hour limit imposed on Personal Access Tokens. Register an organization App with these minimal permissions:
- Organization Permissions > Self-hosted runners: Read & write
- Repository Permissions > Actions: Read-only
- Repository Permissions > Metadata: Read-only
- Repository Permissions > Checks: Read-only
Note your App ID and Installation ID, then download the private key (.pem). Store these values in a Kubernetes secret under the controller’s namespace:
# Create the namespace for ARC
kubectl create namespace arc-systems
# Store credentials in a Secret
kubectl create secret generic arc-github-app \
--namespace=arc-systems \
--from-literal=github_app_id="348210" \
--from-literal=github_app_installation_id="42981045" \
--from-file=github_app_private_key=/path/to/arc-app.private-key.pem
2. Deploy the ARC Operator
Install the controller manager from GitHub’s OCI registry. This chart provides the Custom Resource Definitions and controller logic:
helm install arc \
--namespace arc-systems \
--create-namespace \
oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set-controller
Confirm the controller pod is healthy before moving on:
kubectl get pods -n arc-systems -l app.kubernetes.io/name=gha-runner-scale-set-controller
Configuration
With the operator running, configure an autoscaling runner scale set. The scale set chart manages a lightweight listener pod that monitors GitHub’s job queue and spins up ephemeral runners.
Deploy the Autoscaling Runner Set
First, write a runner-values.yaml file with explicit autoscaling boundaries and container resource targets:
githubConfigUrl: "https://github.com/your-org"
githubConfigSecret: "arc-github-app"
## Autoscaling constraints
minRunners: 0
maxRunners: 30
runnerGroup: "k8s-infra-runners"
template:
spec:
containers:
- name: runner
image: ghcr.io/actions/actions-runner:latest
command: ["/home/runner/run.sh"]
resources:
requests:
cpu: "1000m"
memory: "2Gi"
limits:
cpu: "2000m"
memory: "4Gi"
Install the runner scale set chart using those custom values:
helm install k8s-runner-set \
--namespace arc-runners \
--create-namespace \
-f runner-values.yaml \
oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set
Setting minRunners: 0 ensures pods drop to zero when the queue clears. When a job tags runs-on: k8s-runner-set, the listener detects the queued event and requests a pod from the Kubernetes scheduler.
Targeting the Runner in Workflows
Direct your pipeline jobs to the scale set name in .github/workflows/ci.yml:
name: Integration Tests
on:
push:
branches: [ main ]
pull_request:
jobs:
test:
runs-on: k8s-runner-set
steps:
- name: Checkout Source Code
uses: actions/checkout@v4
- name: Inspect Runner Environment
run: |
echo "Running on host: $(hostname)"
free -m
nproc
Verification & Monitoring
Trigger a workflow dispatch or push a test branch to observe how ARC responds.
1. Inspect Pod Scaling Behavior
Follow pod creation directly in the runner namespace:
kubectl get pods -n arc-runners -w
The listener runs indefinitely. When a job triggers, the runner pod moves from ContainerCreating to Running, completes its task, and terminates automatically:
NAME READY STATUS RESTARTS AGE
k8s-runner-set-754b578d-listener 1/1 Running 0 12m
k8s-runner-set-runner-8f2jc 1/1 Running 0 18s
k8s-runner-set-runner-8f2jc 0/1 Completed 0 52s
k8s-runner-set-runner-8f2jc 0/1 Terminating 0 54s
2. Review Controller Logs During Outages
If workflows stay queued while runners fail to schedule, inspect the listener and controller logs for permission or networking faults:
# Check operator manager logs
kubectl logs -n arc-systems -l app.kubernetes.io/name=gha-runner-scale-set-controller --tail=100
# Check scale set listener logs
kubectl logs -n arc-runners -l app.kubernetes.io/component=runner-scale-set-listener
Most connection errors stem from malformed RSA keys inside the Secret or omitting the Self-hosted runners: Read & write permission in your GitHub App.
3. Handle Node Pressure with Karpenter
Pair ARC with node autoscalers like Karpenter or Cluster Autoscaler. When 20 runners launch at once, pending pods trigger Karpenter to provision spot EC2 instances in under 45 seconds. Once the jobs complete, Karpenter drains and terminates those instances. You get high-throughput CI concurrency during peak business hours without paying for dormant nodes overnight.
