Docker build caching in GitHub Actions: from 12-minute builds to 90 seconds

Every second added to a feedback loop is a tax on engineering velocity. For my algorithmic trading platform, our deployment pipeline was suffering under a massive tax rate. Every commit—even a minor tweak to a trading execution rule or a simple model parameter change—triggered a 12-minute, 43-second build-and-test cycle in GitHub Actions.

Most of this time was spent rebuilding our primary execution container. This container isn’t a lightweight Node.js app; it is a heavy-duty Python environment packed with PyTorch, TA-Lib, custom C++ extensions for fast order serialization, and scientific computation libraries.

A standard docker build command run on a GitHub Actions runner starts fresh every time. Because GitHub provisions a clean virtual machine for each workflow run, local layer caching does not exist.

I set out to crush this bottleneck. After hitting several dead ends with naive caching strategies and disk-serialization hacks, I migrated our pipeline to use BuildKit’s native GitHub Actions cache backend (type=gha). The build time plummeted from over 12 minutes to exactly 91 seconds.

Here is the step-by-step breakdown of how I did it, the dead ends I hit along the way, the configuration code, and the mechanics of why it works.

The Dead Ends: Why Naive Caching Failed

Before arriving at the optimal setup, I tried the intuitive approaches that many developers grab first. They did not work, and in some cases, they actually made our workflows slower.

Dead End 1: Standard actions/cache on /var/lib/docker

My first attempt was to use GitHub’s standard caching action to save and restore the Docker daemon’s internal state directory:

This failed catastrophically. The /var/lib/docker directory is managed by the Docker daemon (dockerd). It contains live sockets, overlay2 storage filesystems, and complex metadata databases. Accessing or writing to this directory while dockerd is running leads to race conditions and file locks.

Even when I stopped the daemon before caching and restarted it afterward, the cache unpack step took upwards of 4 minutes. Worse, we frequently encountered corrupted metadata errors during the cache-restore phase:

Dead End 2: Tarball Export (docker save and docker load)

Next, I attempted to manually export the cached image to a .tar archive, use actions/cache on that archive, and import it before building:

While this was reliable and avoided driver corruption, it hit a massive performance wall: disk I/O bottlenecks.

GitHub Actions Standard Runners (2-core vCPUs) have notoriously slow disk write speeds, often throttling around 100 MB/s. Writing a 3.5 GB trading execution container to a tarball, compressing it, saving it to disk, and then doing the reverse during the restore phase took over 6 minutes. The overhead of compressing and transferring the tarball wiped out almost all the gains of skipping the package installation steps.

The Solution: BuildKit and the gha Cache Backend

To solve this efficiently, we must bypass the host runner’s local filesystem and write cache layers directly to GitHub’s internal cache service API using BuildKit.

BuildKit is the modern engine under the hood of docker build. It supports a variety of cache backends. Rather than relying on local directory hacks or saving monolithic images, BuildKit can serialize individual compilation layers and push them directly to external storage backends.

The most efficient backend for this scenario is the native GitHub Actions cache exporter (type=gha).

flowchart TD
 A["Runner Starts Workflow"] --> B["Setup Docker Buildx Engine"]
 B --> C["Query GitHub Actions Cache API"]
 C -->|"Cache Hit"| D["Pull Cached Layers in Parallel via Network"]
 C -->|"Cache Miss"| E["Execute Local Layer Build"]
 D --> F["Compile New Layer Adjustments"]
 E --> F
 F --> G["Export Updated Layers to GHA Cache API"]
 F --> H["Push Production Image to Registry"]

The magic of type=gha is that it works at the layer level, communicates directly with GitHub’s fast internal API network via HTTP/2, and handles concurrency out of the box. Only the modified or invalidated layers are transferred over the network, minimizing disk I/O on the runner.

The Code

To implement this, you need two tightly integrated components: an optimized, multi-stage Dockerfile and a correctly configured GitHub Actions workflow YAML file.

1. The Optimized Dockerfile

For BuildKit caching to work effectively, your Dockerfile must be structured so that layers that change infrequently (like system packages and Python dependencies) are isolated at the top, while code that changes on every commit is placed at the very bottom.

2. The GitHub Actions Workflow Config

The workflow file configures BuildKit via docker/setup-buildx-action and uses docker/build-push-action to connect our build to the GitHub Action cache.

Behind the Settings: mode=max and Cache Backends

The line cache-to: type=gha,mode=max is where the core optimization lives.

By default, Docker’s cache exporter runs in mode=min. In min mode, only the layers that make up the final stage of the image (the runner stage in our multi-stage build) are cached. The intermediate layers created in our builder stage—such as the compiled C++ extensions and the compiled TA-Lib wheels—are thrown away.

By changing this to mode=max, we instruct BuildKit to serialize and export the cache metadata and layers for every single stage, including intermediate builder stages. On subsequent runs, BuildKit looks at the builder stage and sees that the requirements.txt has not changed. It immediately pulls the pre-built wheels from the GitHub cache, completely skipping the heavy C++ compiling steps and skipping the execution of pip wheel.

Results: Before vs. After

The impact of shifting from no caching (and subsequent failed disk hacks) to BuildKit’s type=gha was immediate. Below is the step-by-step performance profile tracking our production runs.

Before: Pure Build Without Cache

Every run started from scratch. Compiling our specific numerical stack was painfully slow.

After: Build with type=gha Cache Hit

When a cache hit occurs, BuildKit performs dependency evaluation and matches the sha256 checksums of the instructions. Instead of downloading and running compilation, it imports the layers over the fast internal network.

We see a minor time penalty during the export phase of cache hits (1m 13s) because BuildKit checks and matches layer indexes with the GitHub Actions runner cache API. This overhead is trivial compared to the nearly 11-minute savings on compiling.

Lessons Learned & Best Practices

1. Mind the 10GB GitHub Actions Cache Limit: GitHub limits total workflow cache sizes to 10 GB per repository. If your image intermediate steps total 3 GB, you can quickly hit this limit if you keep building multiple PR branches. GitHub automatically evicts older cache blobs, but keeping images trimmed and using multi-stage builds minimizes eviction frequencies.
2. Order Matters: Always put volatile actions (like copying your source code folder COPY src/ /app/src/) as late as possible. If you copy your source code before running a library dependency install, a simple comment change in your app code will invalidate that layer and trigger a complete reinstall of all dependencies downstream.
3. Double Up with Pip Cache Mounts: In the Dockerfile, utilizing --mount=type=cache,target=/root/.cache/pip acts as a local fallback safety net. If a single dependency changes in requirements.txt, the cache layer is invalidated, but pip still uses its local cache directory inside the runner to avoid pulling untouched packages down from PyPI over the open internet.