Jenkins vs GitHub Actions for ML-service CI: what I kept and what I dropped
Two years ago, my team was responsible for maintaining the production deployment pipelines of a suite of real-time machine learning models. These models predict market microstructural alpha and optimize execution routing for a high-frequency trading desk. At the time, our entire continuous integration (CI) infrastructure ran on a self-hosted Jenkins master node backed by a dynamic fleet of AWS EC2 instances.
Every single commit to our codebase triggered a massive Jenkins pipeline. It pulled a 4.2 GB PyTorch model artifact from S3, spun up a g5.4xlarge GPU instance, ran a 10,000-sample regression backtest to check for inference drift, built a Docker container, and pushed it to our private Elastic Container Registry (ECR).
It was a nightmare. We faced constant Jenkins worker starvation, plugin compatibility failures (specifically the AWS EC2 plugin failing to terminate instances under race conditions), and an idle-GPU bill averaging $3,200 a month just for the CI pipeline. Our average PR feedback loop was 42 minutes. If two developers pushed commits simultaneously, the pipeline ground to a halt.
To fix this, we decided to migrate our entire ML pipeline CI infrastructure. But instead of blindly migrating everything to GitHub Actions, we ended up with a hybrid model. Here is exactly what we dropped from Jenkins, what we kept, and how we structured our new GitHub Actions workflow to run fast, deterministic ML service testing.
The Problem: Why Pure Jenkins Failed Our ML Pipeline
Our legacy Jenkins setup suffered from three fatal design flaws that are highly common in machine learning engineering pipelines:
- Stateful Agent Bloat: To avoid downloading 4.2 GB PyTorch models and massive Docker layers on every run, we kept our Jenkins EC2 workers alive with warm-mounted EBS volumes. Over time, these volumes accumulated dangling Docker images, outdated model weights, and corrupted Python virtual environments. Our builds became non-deterministic: a test that passed on “Worker A” would fail on “Worker B” because of an outdated local CUDA driver or a lingering
.pycfile. - The “Plugin Hell” Tax: Jenkins relies on a delicate web of community plugins to interact with GitHub and AWS. Upgrading the Jenkins core frequently broke our OpenID Connect (OIDC) authentication with AWS, halting all deployments for hours while we rolled back the Jenkins master.
- Monolithic Blocking Steps: Our Jenkinsfile ran sequentially. We were running Python lints (
flake8,black --check) on the exact same expensive GPU runner that we used for running inference validation. We were spending $1.62 per hour on a GPU node just to check if someone missed a trailing comma in a configuration file.
Our initial instinct was to migrate 100% of the workload to GitHub Actions-hosted runners. However, we quickly hit the hard realities of GitHub-hosted infrastructure:
* Network Throttling: Downloading large model files from our private AWS S3 buckets into GitHub-hosted runners was painfully slow. S3-to-ECR speeds within the same AWS region are virtually instantaneous; S3-to-GitHub network egress is throttled and expensive.
* GPU Availability & Cost: GitHub-hosted GPU runners are highly expensive and lacked the custom CUDA/cuDNN driver configurations we needed for our low-latency inference runtimes.
The Hybrid Approach: What We Kept and What We Dropped
To solve these constraints, we split the responsibilities:
- What we dropped from Jenkins: We completely removed Jenkins from our Pull Request (PR) validation, code linting, unit testing, and Docker image build steps. GitHub Actions now manages 100% of the developer feedback loop.
- What we kept in Jenkins: We kept Jenkins only as a scheduled scheduler for our heavy, offline model training and historical dataset generation pipelines. Jenkins is excellent at running 12-hour cron-like jobs that run deep inside our on-premise bare-metal servers, where we have unrestricted access to local high-performance storage arrays and massive GPU clusters that we cannot expose to the public GitHub API.
- The Bridge (Self-Hosted Ephemeral Runners): For the critical ML regression tests that require both GitHub integration and GPU access, we deployed the Kubernetes-based
actions-runner-controller(ARC) on our own AWS EKS cluster. This gives us ephemeral, clean, GPU-enabled runners that spin up in our VPC, run the test, and immediately terminate.
Here is the high-level data and control flow of our modernized ML-service CI:
flowchart TD A["Code Push to GitHub"] --> B["GitHub Actions Runner"] B -->|"1. Fast Checks"| C["Linting & Unit Tests (CPU)"] B -->|"2. Dispatch Trigger"| D["Self-Hosted K8s GPU Runner"] D -->|"3. Stream Weights"| E["AWS S3 Bucket"] D -->|"4. Run Regression"| F["Inference Validation & Drift Test"] F -->|"5. Return Code"| B B -->|"6. Push Image"| G["AWS ECR"]
The Code: Our Production GitHub Actions and Pytest Suite
Below is the complete, production-grade GitHub Actions workflow we designed to run this pipeline. It uses AWS OIDC role assumption to avoid hardcoding long-lived secrets, runs parallel linting and unit testing on lightweight GitHub-hosted runners, and triggers our self-hosted GPU runner only when changes occur in our core model code.
1. The GitHub Actions Workflow (.github/workflows/ml-ci.yml)
on:
push:
branches: [ main ]
pull_request:
branches: [ main ]
permissions:
id-token: write
contents: read
jobs:
static-analysis:
runs-on: ubuntu-latest
steps:
– name: Checkout Code
uses: actions/checkout@v4
– name: Set up Python 3.10
uses: actions/setup-python@v5
with:
python-version: "3.10"
cache: "pip"
– name: Install Linting Tools
run: |
python -m pip install –upgrade pip
pip install ruff black mypy
– name: Run Ruff Linter
run: ruff check src/
– name: Check Formatting
run: black –check src/ tests/
– name: Run Static Type Checker
run: mypy –ignore-missing-imports src/
unit-tests:
runs-on: ubuntu-latest
needs: static-analysis
steps:
– name: Checkout Code
uses: actions/checkout@v4
– name: Set up Python 3.10
uses: actions/setup-python@v5
with:
python-version: "3.10"
cache: "pip"
– name: Install Dependencies
run: |
python -m pip install –upgrade pip
pip install -r requirements-test.txt
– name: Run Unit Tests with Mocked Models
run: |
pytest tests/unit -v –durations=10
regression-gpu-test:
runs-on: self-hosted-gpu-runner-set
needs: unit-tests
steps:
– name: Checkout Code
uses: actions/checkout@v4
– name: Configure AWS Credentials
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::112233445566:role/github-actions-ci-role
aws-region: us-east-1
– name: Log in to Amazon ECR
id: login-ecr
uses: aws-actions/amazon-ecr-login@v2
– name: Setup Dynamic Cache Directory
run: |
mkdir -p /mnt/fast-local-nvme/model-cache
echo "MODEL_CACHE_DIR=/mnt/fast-local-nvme/model-cache" >> $GITHUB_ENV
– name: Run Deep Inference Regression Tests
run: |
# Inject cache directory to prevent redundant S3 pulls
export HF_HOME=${{ env.MODEL_CACHE_DIR }}
export MODEL_S3_URI="s3://trading-desk-production-models/alpha-v4/weights.pt"
python -m pytest tests/regression/test_inference_drift.py -v -s
build-and-push-image:
runs-on: ubuntu-latest
needs: regression-gpu-test
if: github.ref == 'refs/heads/main'
steps:
– name: Checkout Code
uses: actions/checkout@v4
– name: Configure AWS Credentials
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::112233445566:role/github-actions-ci-role
aws-region: us-east-1
– name: Log in to Amazon ECR
id: login-ecr
uses: aws-actions/amazon-ecr-login@v2
– name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
– name: Build and Push Production Container
uses: docker/build-push-action@v5
with:
context: .
file: ./docker/Dockerfile.prod
push: true
tags: |
112233445566.dkr.ecr.us-east-1.amazonaws.com/alpha-predictor:${{ github.sha }}
112233445566.dkr.ecr.us-east-1.amazonaws.com/alpha-predictor:latest
cache-from: type=gha
cache-to: type=gha,mode=max
2. The Regression Testing Suite (tests/regression/test_inference_drift.py)
This is the exact structure of the script we use to download our models inside our self-hosted runner and run regression tests. It uses local fast NVMe disk caching to ensure that if a model version hasn’t changed, we do not waste time downloading it over the network.
import sys
import boto3
import torch
import numpy as np
import pytest
from pathlib import Path
# Force CPU fallback if CUDA isn't configured correctly on the host
DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")
def download_model_from_s3(s3_uri: str, local_cache_dir: str) -> Path:
"""
Downloads model weights from S3 to a local fast storage path with basic validation.
"""
s3 = boto3.client("s3")
bucket = s3_uri.split("/")[2]
key = "/".join(s3_uri.split("/")[3:])
file_name = key.split("/")[–1]
local_path = Path(local_cache_dir) / file_name
# If the file already exists locally, skip downloading
if local_path.exists():
print(f"Cache hit: Found {local_path} locally. Skipping download.")
return local_path
print(f"Downloading {s3_uri} to {local_path}…")
local_path.parent.mkdir(parents=True, exist_ok=True)
# Download with progress printed to logs
s3.download_file(bucket, key, str(local_path))
return local_path
@pytest.fixture(scope="module")
def model_instance():
"""
Loads the production model to the designated compute device.
"""
s3_uri = os.getenv("MODEL_S3_URI")
cache_dir = os.getenv("MODEL_CACHE_DIR", "/tmp/model-cache")
if not s3_uri:
pytest.fail("MODEL_S3_URI environment variable must be specified for regression tests.")
model_file_path = download_model_from_s3(s3_uri, cache_dir)
# Initialize our internal neural network class structure
# Assuming standard PyTorch Jit traced or State Dict model
try:
model = torch.jit.load(str(model_file_path), map_location=DEVICE)
model.eval()
except Exception as e:
pytest.fail(f"Failed to load compiled PyTorch model: {str(e)}")
return model
def test_gpu_acceleration():
"""
Asserts that the self-hosted runner is utilizing a healthy GPU device.
"""
assert torch.cuda.is_available(), "CUDA device not available! Regression tests must run on GPU."
print(f"Running regression tests on device: {torch.cuda.get_device_name(0)}")
def test_inference_stability(model_instance):
"""
Asserts model outputs are stable by passing a synthetic batch of high-frequency orderbook states.
Checks that outputs do not fall outside the statistical bounds established in backtesting.
"""
# 64 samples, representing 20 depth features and 5 lag steps (64, 5, 20)
np.random.seed(42)
synthetic_input = np.random.normal(loc=0.0, scale=1.0, size=(64, 5, 20))
input_tensor = torch.tensor(synthetic_input, dtype=torch.float32).to(DEVICE)
with torch.no_grad():
predictions = model_instance(input_tensor)
# Move outputs back to CPU for evaluation
output_array = predictions.cpu().numpy()
assert not np.isnan(output_array).any(), "NaN values found in model inference outputs!"
assert not np.isinf(output_array).any(), "Infinite values found in model inference outputs!"
# Expectation: predictions represent alpha return values.
# Must remain within plausible microstructural bands (-5.0% to +5.0% per frame)
mean_prediction = np.mean(output_array)
std_prediction = np.std(output_array)
print(f"Inference Stats -> Mean: {mean_prediction:.6f}, Std: {std_prediction:.6f}")
assert –0.05 <= mean_prediction <= 0.05, f"Mean prediction {mean_prediction} is outside expected trade limits."
assert std_prediction > 0.0, "Model is returning static output across varying inputs."
Results: The Impact of Decoupling
The shift away from monolithic Jenkins CI runs to a lean, GitHub Actions-driven pipeline yielded immediate and massive improvements across our deployment cycle.
| Metric | Legacy Jenkins Setup | Modern GitHub Actions + Hybrid ARC Setup | Change |
|---|---|---|---|
| Average PR Feedback Loop | 42 minutes | 6.5 minutes | -84.5% |
| Monthly Compute Cost (CI) | $3,210.00 | $412.00 | -87.1% |
| Pipeline Failure Rate (Flakiness) | 12.4% (Resource exhaustion/leaks) | < 0.2% | -98.3% |
| Developer Onboarding Time | Write Jenkinsfile DSL + debug plugins | Standard YAML schema in git repo | Hours to minutes |
By shifting lints and unit tests to standard GitHub-hosted virtual machines, we parallelized our pipeline stages. Developers now find out if their formatting or standard unit tests failed within 90 seconds of pushing their branch, rather than waiting for an expensive GPU machine to scale up in AWS, download the Docker image base layers, and crash because of a missing colon in a python docstring.
Our compute costs plummeted because of our Kubernetes self-hosted setup. Rather than keeping a warm EC2 instance pool waiting for jobs, the actions-runner-controller dynamically provisions pod instances using AWS Karpenter. Once the regression test finishes, Karpenter immediately reclaims the node if no other jobs are pending.
Lessons Learned: What I Deeply Regret and What I Got Right
1. The Trap of “Docker-in-Docker” in Self-Hosted Runners
When we first set up our self-hosted runner group on Kubernetes to execute regression tests, we tried to build our Docker containers directly inside the self-hosted pods using Docker-in-Docker (DinD). This required running the pods in privileged mode, which violated our security compliance standards and led to massive local cache corruption issues.
* What we did instead: We separated the regression test from the image build. The regression test runs purely in a raw Python environment configured with CUDA. The actual Docker image build is executed on standard GitHub Actions runners using docker/build-push-action backed by remote gha cache exporters, completely avoiding privileged container executions.
2. Never Query AWS Secrets Manager in Loop Steps
Initially, our regression tests pulled AWS parameters and API keys dynamically during the run execution. This was fine on our dedicated Jenkins server, which was constantly authenticated, but inside ephemeral K8s runner pods, it triggered rapid API rate limits from AWS Secrets Manager when multiple PRs ran concurrently. We shifted to using IAM Roles for Service Accounts (IRSA) on the Kubernetes pod and bound our environment secrets using GitHub Actions secrets directly injected into the runner env vars at startup.
3. Decouple Code CI from Data Pipelines
The single most important decision we made was accepting that code CI is not a data pipeline.
We stopped trying to make GitHub Actions train our models. If a pipeline takes longer than 15 minutes, it belongs in a dedicated orchestrator. Jenkins excels at orchestrating these long, heavy, scheduled tasks that run on deep on-prem databases. GitHub Actions excels at keeping developers fast, validated, and safely automated. Splitting them saved our budget and restored developer sanity.