Local Kubernetes dev environments: k3d, Tilt, and the inner loop that finally stuck

The worst part of building distributed execution systems is the time wasted waiting for Docker images to compile, push, pull, and deploy. When you are writing latency-sensitive algorithmic trading adapters or real-time event-driven engines, you cannot rely solely on unit tests. You need to verify how your service behaves when interacting with an in-cluster message broker, a mock exchange API, and a database.

For years, my loop looked like this:
1. Make a one-line change in a Python or Go service.
2. Run docker build -t myregistry.local/execution-service:latest ..
3. Wait 90 seconds for layers to rebuild.
4. Run docker push myregistry.local/execution-service:latest.
5. Run kubectl rollout restart deployment execution-service.
6. Wait 45 seconds for Kubernetes to pull the image, fail its liveness probe, terminate the old pod, and start the new one.
7. Realize I made a syntax error. Repeat.

A three-minute feedback loop is a productivity killer. It forces you to batch your work, lose focus, and spend half your day browsing hacker news while waiting for bash scripts to finish.

I tried Minikube, but it chewed through my MacBook’s battery and felt sluggish. I tried raw Docker Compose, but it failed to replicate Kubernetes-specific patterns like init-containers, ingress rules, and configmap mounts.

The combination that finally fixed this is k3d combined with Tilt. My inner loop went from 180 seconds to under 2 seconds. This post goes through the exact configuration, code, and architecture I use to run high-throughput services locally.

The local Kubernetes architecture

To make local kubernetes development fast, you must avoid rebuilding whole container layers and bypass the remote registry entirely.

k3d runs lightweight k3s (a minimal Kubernetes distribution by Rancher) inside Docker containers. It is fast to spin up, uses very little memory, and supports local registries out of the box. Tilt monitors your local workspace, detects code modifications, and live-injects those changes directly into the running containers without restarting them, unless a dependency change requires a full rebuild.

flowchart TD
 dev["Developer Machine"]
 tilt["Tilt Engine"]
 k3d["k3d Cluster"]
 reg["Local Registry"]
 app["App Container"]
 tilt -->|"Watches File System"| dev
 tilt -->|"Builds & Pushes"| reg
 reg -->|"Pulls Image"| k3d
 tilt -->|"Live Syncs Code"| app
 k3d -->|"Hosts"| app

Setting up the infrastructure with k3d

First, we need a reproducible way to spin up a local Kubernetes cluster with a built-in registry and correct port mapping. Avoid doing this via imperitive CLI commands; use a configuration file so anyone on your team can spin up the identical environment with a single command.

Here is the k3d-config.yaml I use:

This configuration achieves three things:
1. It disables the default Traefik ingress controller because I prefer to use standard NGINX or let Tilt manage ingress paths directly.
2. It sets up a dedicated local container registry running at 127.0.0.1:5005.
3. It maps port 8080 on your host machine to port 80 on the k3d load balancer, allowing you to access cluster services locally without running kubectl port-forward manually every time.

To spin up this cluster, run:

Output:

The Target Application

Let’s build a fast, lightweight mock trading execution engine in Python using FastAPI. The service will ingest buy/sell orders, simulate latency, and log the executions.

Here is the app.py:

To make Tilt’s live update engine work smoothly, write a standard Dockerfile that sets up a clean environment but does not compile production-only optimizations that prevent on-the-fly execution changes.

Next, write the standard Kubernetes deployment manifest (deployment.yaml). This manifest keeps things simple, but notice that we will use a specific tag image that our local registry supports.

Configuring the Tiltfile

The Tiltfile is written in Starlark, a dialect of Python. It tells Tilt how to build your images, track changes on your local filesystem, and update the cluster.

If you modify app.py, we do not want Tilt to rebuild the entire container image. Instead, we want Tilt to copy the changed file straight into the running container container filesystem and signal the application process (or let uvicorn auto-reload catch it).

Here is the Tiltfile:

Running the setup

With the k3d cluster running, boot up your Tilt environment:

Tilt will spin up its interactive terminal UI and start a web console at http://localhost:10350.

Now send an execution request to verify it is working:

Output response:

Test the Live Update Loop

Now let’s test how fast Tilt synchronizes changes. Let’s modify our execution pipeline logic inside app.py to add validation constraints (e.g., rejecting negative values or quantities) without rebuilding the container.

In your local IDE, modify app.py directly by inserting these lines at the beginning of place_order:

The moment you save the file, Tilt’s filesystem listener captures the change. Look at your terminal output or Tilt web UI:

Verify that the runtime changes were instantly adopted without restarting the pod:

The response is instantaneous:

No image compilation, no docker push, no kubernetes scheduling delays. The change deployed in less than half a second.

Lessons learned from running k3d + Tilt at scale

This setup is fast, but things can break if you don’t scale the configurations properly. Here are the three most common failures I ran into while tuning local environments with hundreds of microservices.

1. Inotify limits on Linux

If you are developing on Linux (or running docker inside a Linux VM), you might get file-watching errors when your workspace gets large:

This happens because the system’s fs.inotify.max_user_watches value is set too low. Fix this by adding the following to your host’s /etc/sysctl.conf:

Apply the changes immediately with sudo sysctl -p.

2. Live Update vs Pip / Dependency updates

If you modify your requirements.txt or go.mod file, a simple hot reload of the code file is not enough; you need to trigger a full build because a dependency is missing. You can configure Tilt to handle this dynamically:

Adding trigger targets lets Tilt know that modifying dependencies should bypass live updates and execute a full, clean Docker build.

3. Avoiding State Desynchronization

If your local application holds persistent connections (e.g. gRPC streams, WebSocket subscriptions to tick feeds), updating files via live update can leave stale connections hanging or cause unexpected edge-case errors inside the engine. If your application does not robustly handle dynamic file hot-reloading at runtime, add a fallback trigger to restart the process inside the container: