Earthly Deployment: Scaling Distributed Build Caches for Container-Native CI/CD

The “Works on My Machine” Nightmare

I remember sitting in a dark office at 2 AM, staring at a Jenkins pipeline that refused to turn green. Locally, the Go binary compiled in 14 seconds, tests passed, and the Docker image was ready for deployment. On the CI runner, it crashed with a cryptic error about a missing libssl header. It turned out my local Ubuntu machine had libssl-dev pre-installed, while the Jenkins agent—maintained by a separate platform team—was running a stripped-down Alpine image.

This is the classic environmental drift problem. For decades, we relied on Makefiles or Bash scripts to glue build steps together. Make is a legendary tool for local task execution, but it treats your host like a buffet; it assumes every dependency is already laid out exactly as it likes.

When you move that logic to GitHub Actions or GitLab CI, you often end up writing a 300-line YAML file just to recreate your local environment. This duplication of logic is a maintenance trap. It is the number one reason CI/CD pipelines fail.

Why Traditional Build Scripts Fail in Modern DevOps

After auditing dozens of enterprise projects, I found two structural flaws that make build systems brittle. The first is a Lack of Hermeticity. A build is hermetic if it produces the same bit-for-bit output regardless of the host. Makefiles are rarely hermetic because they leak host environment variables, local file paths, and system-level binary versions. If a developer uses Node v20 while the runner is stuck on v18, the build might pass locally but fail in production without any warning.

The second flaw is Caching Complexity. Standard CI pipelines are slow because they usually start from a blank slate. We try to patch this with tools like actions/cache, but these exist outside the build logic itself. If you update a minor dependency in a 4GB project, the cache invalidation is often too blunt, forcing a full rebuild. My rule of thumb is simple: your build logic must be decoupled from your runner infrastructure to be truly reliable.

Makefile vs. Docker vs. Earthly: A Critical Comparison

Choosing the right tool depends on your isolation needs. Makefiles offer simplicity but zero isolation. Docker-in-Docker (DinD) provides isolation but is notoriously difficult to cache efficiently in CI environments. Every layer needs to be pushed and pulled manually, which often negates the time saved by caching.

Earthly fills this gap. It acts as a hybrid between a Dockerfile and a Makefile. It adopts Docker’s familiar syntax (FROM, RUN, COPY) but introduces Make-style targets. Every step in an Earthly build executes inside a container. If it works on your MacBook Pro, it will work exactly the same way on a Linux CI agent. Earthly also manages distributed caching natively. It shares build layers between your local terminal and your CI runners without requiring you to manage complex cache keys.

The Earthly Approach: Build Once, Run Everywhere

Getting started is straightforward. Earthly only requires a Docker daemon to be running on the host machine. Once you install the CLI, your project’s build logic moves into a single Earthfile.

# Install Earthly on Linux
sudo /bin/sh -c 'wget https://github.com/earthly/earthly/releases/latest/download/earthly-linux-amd64 -O /usr/local/bin/earthly && chmod +x /usr/local/bin/earthly'

Consider a Python application. In a standard setup, you would manage a requirements.txt and a separate Makefile. In Earthly, you define targets that represent specific lifecycle stages.

# Earthfile
VERSION 0.8

FROM python:3.11-slim
WORKDIR /app

setup:
    COPY requirements.txt .
    RUN pip install -r requirements.txt

test:
    FROM +setup
    COPY . .
    RUN pytest . 

build:
    FROM +setup
    COPY . .
    RUN python setup.py build
    SAVE ARTIFACT ./dist /dist AS LOCAL ./dist

The test target depends on +setup. If you modify a line of code but leave requirements.txt untouched, Earthly skips the pip install step. It calculates the layer checksum and pulls the result from the cache. Because this happens inside a container, the Python version on your host machine becomes irrelevant.

Practical Implementation: Moving from Makefile to Earthfile

Do not delete your Makefile immediately during a migration. Instead, use it as a thin wrapper. This allows developers to keep using make test while Earthly handles the actual heavy lifting in the background.

Satellite Builds are a game-changer for large teams. If your local machine is struggling under the load of a microservices build, you can point Earthly to a “Satellite”—a dedicated 32-core build server. The CLI transparently offloads the build context to this remote server, executes the build, and streams the logs back to your terminal. It gives every engineer the power of a data center for their local development.

# Running a target locally but with remote caching
earthly --remote-cache=my-registry.com/project-cache +build

Leveraging Distributed Caching for Faster CI/CD

The real power of Earthly is felt during CI integration. You no longer need 50 lines of YAML to configure Node, Python, or Go environments. Your GitHub Action configuration usually shrinks to a single, readable step.

name: CI
on: [push]
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Install Earthly
        uses: earthly/actions-setup@v1
      - name: Run Build
        run: earthly --push +all

Using the --push flag tells Earthly to export artifacts to your registry and upload the build cache simultaneously. When a teammate runs earthly +all five minutes later, their machine downloads the exact cache layers generated by the CI runner. We have seen build times for monolithic applications drop from 15 minutes to under 120 seconds by leveraging this shared cache.

Final Thoughts on Build Engineering

Adopting Earthly requires a mental shift. You must view your build as a directed acyclic graph (DAG) of isolated containers rather than a linear sequence of shell commands. The initial setup takes more effort than a quick Bash script, but the long-term reliability is undeniable.

Environment-specific bugs disappear. Inconsistent local setups become a thing of the past. If you manage complex pipelines, start small: pick your most unreliable build step, wrap it in an Earthly target, and watch the stability improve. Once you experience a build that truly runs identical across every machine, you will never go back to raw Makefiles.