GitHub Actions self-hosted runners on Kubernetes: cost, caching, and cleanup
At my algorithmic trading firm, our execution algorithms and alpha research pipelines depend on rapid feedback loops. If a quantitative researcher has to wait forty minutes for a backtesting Docker image to build and pass integration tests, our research velocity stalls.
Six months ago, our GitHub Actions bill was spiraling. We were heavily utilizing GitHub-hosted runners, particularly the larger enterprise instances (32-core VMs) to parallelize our C++ compilations and PyTorch model packaging. Our monthly ci cost was hovering around $14,200. Worse, we were constantly hitting the limits of GitHub’s hosted cache, leading to frequent cache misses that forced full dependency rebuilds.
I migrated our entire CI/CD infrastructure to self-hosted runners running on Amazon Elastic Kubernetes Service (EKS) using the Actions Runner Controller (ARC). This is the exact playbook of how we built an autoscaling, self-cleaning runner fleet that slashed our monthly ci cost by 81% while dropping our median build times from 18 minutes to under 4 minutes.
The architectural bottleneck
Our initial migration to self-hosted runners on Kubernetes was a disaster. We initially deployed a static stateful set of runners. This naive approach introduced three major issues:
- State Poisoning: Leftover files from previous builds would pollute the workspace of subsequent runs. A failing test in build #104 could leave a corrupted database file on the runner, causing build #105 to fail mysteriously.
- Compute Underutilization: We had to provision for peak load. During high-intensity periods (e.g., right before market close when researchers push their daily strategies), the runners were fully saturated. At night, we paid for idle EC2 instances sitting completely empty.
- Slow I/O: Compiling our backtester requires writing hundreds of thousands of small files. Standard network-attached EBS volumes (gp3) choked on IOPS limits, rendering our expensive CPU cores idle while waiting on disk I/O.
To solve this, we redesigned the platform around ephemeral runners, Karpenter-driven spot instance provisioning, and local NVMe caching.
flowchart TD A["GitHub Event Trigger"] --> B["Actions Runner Controller"] B -->|"Scale Request"| C["Karpenter Autoscaler"] C -->|"Provision Spot VM"| D["EKS Node with NVMe"] D -->|"Run Ephemeral Pod"| E["Runner Pod"] E -->|"Read/Write Cache"| F["Local NVMe HostPath"]
Infrastructure Blueprint: ARC and Karpenter
The modern way to run GitHub Actions on Kubernetes is through the new Actions Runner Controller (ARC) autoscaling runner scale sets. It deprecates the older, stateful operator model in favor of an architecture that listens directly to GitHub’s long-polling event stream.
Here is the Helm configuration we use to deploy the ARC autoscaling runner scale set. This file handles the critical security and lifecycle hooks needed to ensure pods are discarded immediately after a single job execution.
values.yaml for gha-runner-scale-set
githubConfigSecret: “github-app-token-secret”
minRunners: 0
maxRunners: 50
controllerServiceAccount:
namespace: arc-systems
name: arc-gha-rs-controller
template:
spec:
containers:
– name: runner
image: “ghcr.io/our-quant-org/gha-runner:v1.2.4”
command: [“/home/runner/run.sh”]
env:
– name: ACTIONS_RUNNER_REQUIRE_JOB_CONTAINER
value: “false”
– name: DOCKER_HOST
value: “unix:///run/docker/docker.sock”
volumeMounts:
– name: work
mountPath: /home/runner/_work
– name: docker-sock
mountPath: /run/docker
– name: nvme-cache
mountPath: /home/runner/.cache
services:
– name: dind
image: “docker:24.0.7-dind”
securityContext:
privileged: true
env:
– name: DOCKER_TLS_CERTDIR
value: “”
volumeMounts:
– name: docker-sock
mountPath: /run/docker
– name: docker-storage
mountPath: /var/lib/docker
– name: nvme-cache
mountPath: /var/lib/docker/cache
volumes:
– name: work
emptyDir: {}
– name: docker-sock
emptyDir: {}
– name: docker-storage
emptyDir: {}
– name: nvme-cache
hostPath:
path: /mnt/karpenter/nvme/gha-cache
type: DirectoryOrCreate
This configuration achieves strict isolation by pairing each runner container with a sidecar Docker-in-Docker (dind) container sharing a UNIX socket (docker-sock) via an emptyDir volume. This bypasses the need to mount the host’s /var/run/docker.sock, securing the underlying EKS node from container escape vectors.
To dynamically scale this infrastructure, we use Karpenter rather than the standard Cluster Autoscaler. Karpenter binds directly to the AWS EC2 fleet APIs, provisioning instances in under 15 seconds compared to the 2 minutes typically required by Managed Node Groups.
Here is our Karpenter NodePool manifest, specifically tuned to lease high-performance, cost-effective AMD EPYC spot instances equipped with local NVMe SSDs (the c6id and m6id families):
karpenter-nodepool.yaml
kind: NodePool
metadata:
name: gha-runners
spec:
template:
spec:
requirements:
– key: “karpenter.sh/capacity-type”
operator: In
values: [“spot”]
– key: “kubernetes.io/arch”
operator: In
values: [“amd64”]
– key: “node.kubernetes.io/instance-type”
operator: In
values: [“c6id.2xlarge”, “c6id.4xlarge”, “m6id.2xlarge”, “m6id.4xlarge”]
nodeClassRef:
name: gha-node-class
disruption:
consolidationPolicy: WhenUnderutilized
expireAfter: 72h
—
apiVersion: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
metadata:
name: gha-node-class
spec:
amiFamily: Bottlerocket
role: “KarpenterNodeRole-GHA”
subnetSelectorTerms:
– tags:
karpenter.sh/discovery: “eks-prod”
securityGroupSelectorTerms:
– tags:
aws:eks:cluster-name: “eks-prod”
blockDeviceMappings:
– deviceName: /dev/xvda
ebs:
volumeSize: 50Gi
volumeType: gp3
iops: 3000
throughput: 125
userData: |
# RAID0 setup script for multiple ephemeral local NVMe disks to maximize IOPS
[settings.kernel]
modules = [“raid0”]
[settings.bootstrap]
extra-user-data = “””
#!/usr/bin/env bash
set -Eeuo pipefail
DEVICES=($(lsblk -d -n -o NAME,MODEL | grep “Amazon EC2 NVMe Instance Storage” | awk ‘{print “/dev/”$1}’))
if [ ${#DEVICES[@]} -gt 0 ]; then
mkdir -p /mnt/karpenter/nvme
if [ ${#DEVICES[@]} -eq 1 ]; then
mkfs.ext4 -F ${DEVICES[0]}
mount ${DEVICES[0]} /mnt/karpenter/nvme
else
mdadm –create –verbose /dev/md0 –level=0 –raid-devices=${#DEVICES[@]} “${DEVICES[@]}”
mkfs.ext4 -F /dev/md0
mount /dev/md0 /mnt/karpenter/nvme
fi
mkdir -p /mnt/karpenter/nvme/gha-cache
chmod 777 /mnt/karpenter/nvme/gha-cache
fi
“””
Solving the Caching Puzzle
If your self-hosted runners download dependencies from scratch on every run, you are wasting time and bandwidth. Standard actions/cache stores tarballs in GitHub’s cloud, downloading them over the internet during run-time. This is too slow for 10GB+ compiler caches and model weights.
Instead, we map a local NVMe directory /mnt/karpenter/nvme/gha-cache directly into our ephemeral runner pods via a hostPath volume.
To prevent concurrent workflows from overwriting each other’s cache files on the same host, we leverage BuildKit’s native caching mechanisms combined with dynamic sub-directory isolation. Here is our standardized Dockerfile that compiles our trading modules using compiler cache mounts:
Dockerfile.builder
FROM ubuntu:22.04 AS builder
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y \
build-essential \
cmake \
ccache \
clang \
git \
&& rm -rf /var/lib/apt/lists/*
ENV CCACHE_DIR=/workspace/.ccache
ENV CC=/usr/bin/clang
ENV CXX=/usr/bin/clang++
WORKDIR /workspace
# Mount ccache and apt cache directories as persistent cache mounts
RUN –mount=type=cache,target=/workspace/.ccache \
–mount=type=cache,target=/var/cache/apt,sharing=locked \
–mount=type=bind,source=.,target=/workspace \
mkdir -p build && \
cd build && \
cmake -DCMAKE_BUILD_TYPE=Release .. && \
make -j$(nproc)
In our pipeline workflow files, we run the builds utilizing local registries hosted within our VPC (Amazon ECR). This avoids public internet traffic completely:
ci-pipeline.yml
on:
push:
branches: [ main ]
jobs:
build-and-test:
runs-on: arc-runner-set-trading
steps:
– name: Checkout Code
uses: actions/checkout@v4
– name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
with:
driver: kubernetes
driver-opts: |
image=moby/buildkit:buildx-stable-1
namespace=arc-systems
– name: Log in to Amazon ECR
run: |
aws ecr get-login-password –region us-east-1 | \
docker login –username AWS –password-stdin 111122223333.dkr.ecr.us-east-1.amazonaws.com
– name: Build and Push with Registry Cache
uses: docker/build-push-action@v5
with:
context: .
file: Dockerfile.builder
push: true
tags: 111122223333.dkr.ecr.us-east-1.amazonaws.com/trading-engine:${{ github.sha }}
cache-from: type=registry,ref=111122223333.dkr.ecr.us-east-1.amazonaws.com/trading-engine:buildcache
cache-to: type=registry,ref=111122223333.dkr.ecr.us-east-1.amazonaws.com/trading-engine:buildcache,mode=max
Handling the Garbage: Automated Cleanup
When running hundreds of ephemeral builds daily, Docker-in-Docker caches and local NVMe host paths quickly accumulate junk. If a node’s local disk fills up to 90%, Kubernetes marks the node with DiskPressure taint, preventing further pods from scheduling on it.
To solve the orphaned storage problem, we run a Kubernetes DaemonSet on our runner nodes that executes a aggressive cleanup script every 30 minutes.
cache-cleaner-daemonset.yaml
kind: DaemonSet
metadata:
name: gha-node-cleaner
namespace: arc-systems
spec:
selector:
matchLabels:
app: gha-node-cleaner
template:
metadata:
labels:
app: gha-node-cleaner
spec:
tolerations:
– key: “karpenter.sh/capacity-type”
operator: Equal
value: “spot”
effect: NoSchedule
containers:
– name: cleaner
image: alpine:3.18
securityContext:
privileged: true
volumeMounts:
– name: host-nvme
mountPath: /host/nvme
– name: host-docker-root
mountPath: /host/var/lib/docker
command: [“/bin/sh”, “-c”]
args:
– |
while true; do
echo “Starting node storage sweeping…”
# Clean up host-mounted workspace caches older than 3 days
find /host/nvme/gha-cache/ -mindepth 1 -maxdepth 2 -mtime +3 -exec rm -rf {} \;
# Check disk usage on local NVMe
DISK_USAGE=$(df -h /host/nvme | awk ‘NR==2 {print $5}’ | sed ‘s/%//’)
if [ “$DISK_USAGE” -gt 80 ]; then
echo “Disk usage on NVMe is above 80% ($DISK_USAGE%). Forcing emergency purge…”
# Delete oldest run caches until usage is back below 60%
find /host/nvme/gha-cache/ -mindepth 1 -maxdepth 1 -type d -printf ‘%T+ %p\n’ | \
sort | \
cut -d’ ‘ -f2- | \
while read -r dir; do
rm -rf “$dir”
CURRENT_USAGE=$(df -h /host/nvme | awk ‘NR==2 {print $5}’ | sed ‘s/%//’)
if [ “$CURRENT_USAGE” -lt 60 ]; then
break
fi
done
fi
echo “Clean completed. Sleeping for 1800s.”
sleep 1800;
done
volumes:
– name: host-nvme
hostPath:
path: /mnt/karpenter/nvme
type: Directory
– name: host-docker-root
hostPath:
path: /var/lib/docker
type: Directory
Real-world telemetry and cost reductions
Prior to implementing this self-hosted architecture, our GitHub enterprise bill broke down to approximately $14,200 per month. By shifting compute to Amazon EC2 Spot Instances managed by ARC and Karpenter, our primary ci cost shifted to standard AWS infrastructure billing.
Here is the exact comparison of our monthly costs pre- and post-migration:
| Expense Category | Old Architecture (GitHub Hosted) | New Architecture (EKS + Spot ARC) |
|---|---|---|
| GitHub Actions Compute Minutes | $14,200 | $0 (included in Enterprise base) |
| AWS EC2 Spot Compute | $0 | $2,180 |
| AWS EKS Control Plane Fee | $0 | $73 |
| EBS & Local NVMe Storage | $0 | $345 |
| AWS Nat Gateway / Data Transfer | $0 | $140 |
| Total Monthly Run-rate | $14,200 | $2,738 |
Build Speed Improvement
The local NVMe caching strategy eliminated internet network latency. Our compile times for the core C++ strategy backtester decreased drastically:
– Pulling dependencies: 3m 45s
– C++ Compile & Linking: 11m 20s
– Post-steps & Upload: 1m 40s
TOTAL TIME: 18m 00s [New Platform – EKS HostPath NVMe + BuildKit] – Pod Provisioning (Karpenter): 0m 12s
– Cache hit verification: 0m 04s
– Dynamic Compilation (ccache warmed): 2m 14s
– Post-steps & ECR push: 0m 42s
TOTAL TIME: 3m 12s
Lessons learned and operational gotchas
Deploying self-hosted runners on Kubernetes is not a “set-and-forget” exercise. We hit several significant failures during our transition:
The API Rate-Limit Ceiling
When we scaled up to 40 concurrent runners, they began polling the GitHub API simultaneously to check for available jobs. GitHub’s rate limiter kicked in, temporarily blocking our entire VPC’s egress IP from communicating with GitHub.
The fix: We transitioned away from using Personal Access Tokens (PATs) and switched to utilizing a GitHub App configuration for authentication in ARC. GitHub Apps receive a significantly higher API rate limit (up to 15,000 requests per hour, scaling up dynamically with repository traffic).
Docker-in-Docker Pod Zombies
If a workflow was cancelled by a developer, the runner pod would terminate immediately, but occasionally the underlying dind sidecar container failed to receive the SIGTERM cleanly. This resulted in zombie containers consuming system CPU resources.
The fix: We added an explicit pre-stop hook in the values.yaml for our runners to gracefully shutdown the docker daemon before the container receives its SIGKILL signal:
preStop:
exec:
command: [“/bin/sh”, “-c”, “docker system prune -af && kill -TERM $(cat /var/run/docker.pid)”]
These configurations successfully optimized our pipelines, creating a fast, cost-effective build environment that keeps our research loops highly efficient.