Multi-environment GitOps: one repo layout that survived dev, staging, and prod

Two years ago, I broke production during a high-volatility trading session because our GitOps repository structure lied to us.

We were running a standard monorepo setup using Helm charts and ArgoCD. To deploy a fix to dev, I merged a pull request that bumped a shared parameter in a parent values file. What I didn’t realize was that our Helm dependency graph contained an implicit cross-environment override. Thirty seconds after the merge, ArgoCD auto-synced the change across all clusters simultaneously. The execution engine lost connection to our broker gateway, triggering a cascade of circuit breakers that cost us $42,000 in unhedged inventory risk before we could manually halt the sync.

We tried splitting into separate repositories per environment. That created a nightmare: PRs had to be cherry-picked across three distinct repos (trading-app-dev, trading-app-staging, trading-app-prod), configuration drifted wildly within weeks, and no one actually knew what was running where.

This post details the exact GitOps repository structure I engineered to solve this problem permanently—a single-repo layout powered by Kustomize and ArgoCD that maintains total environment isolation, eliminates configuration duplication, and guarantees deterministic promotions.

The Core Architecture

The architecture relies on a strict directory hierarchy using Kustomize overlays. We explicitly banned Helm template logic inside our application manifests, reserving Helm exclusively for third-party upstream dependencies (like Redis or Kafka operators). For our own microservices, pure Kustomize base/overlay patterns proved far easier to audit, diff, and dry-run.

Here is the flow of configuration changes from local development to production deployment:

flowchart LR
 dev["Developer PR"] -->|"Merge to main"| base["Base Directory"]
 base -->|"Kustomize Overlay"| devEnv["Dev Cluster"]
 devEnv -->|"ArgoCD Auto-Sync"| devApp["Dev Deployment"]
 base -->|"Image Tag Pin"| stagingEnv["Staging Cluster"]
 stagingEnv -->|"Manual Sync"| stagingApp["Staging Deployment"]
 stagingEnv -->|"Diff Validation"| prodEnv["Prod Cluster"]
 prodEnv -->|"Gated ArgoCD Sync"| prodApp["Prod Deployment"]

Directory Taxonomy

To prevent cross-environment pollution, the repository enforces three hard rules:
1. Bases are immutable per environment release: A base/ directory defines what runs (Deployments, Services, ConfigMap keys), never how it runs (replica counts, CPU limits, external ingress hostnames).
2. Environments are isolated top-level trees: environments/dev, environments/staging, and environments/prod do not reference each other via relative paths. They only reference ../../apps/<app-name>/base.
3. Application manifests and GitOps control manifests live together: ArgoCD Application resources live inside the root bootstrap/ folder, referencing specific paths and commit SSH keys explicitly.

Here is the working directory tree of our layout:

Deep Dive: Base & Overlay Configurations

Let’s look at the actual code driving this setup. We will examine the order-execution engine service, which demands low-latency execution and precise environment-specific overrides.

The Base Definition

The base defines the structural skeleton. Notice that there are no hardcoded environment variables, replica counts, or resource limits here.

The Production Overlay

In production, we lock down image tags using precise SHA digests, scale out replica sets, apply strict resource constraints, and introduce a TopologySpreadConstraint to guarantee multi-AZ availability across our AWS us-east-1 EKS worker nodes.

ArgoCD Bootstrap via the App-of-Apps Pattern

To tie everything together without touching kubectl on target clusters, we use ArgoCD’s “App-of-Apps” paradigm. The root bootstrapper points to a single directory containing ArgoCD manifest definitions for each environment.

Now, we define environment-level declarative control policies inside bootstrap/apps/. Notice the fundamental difference in sync strategies: dev is aggressive and automatic, while prod has automated disabled, requiring manual execution via ArgoCD CLI or UI after automated diff checks pass in CI.

Automated Validation Pipeline

A massive multi-environment repository easily breaks if developers mess up relative path traversal in Kustomize files or commit malformed YAML. We run strict automated checks on every pull request using a Python validation script integrated into GitHub Actions.

This script ensures that:
1. Every Kustomize overlay builds cleanly without schema errors.
2. Production overlays do not contain mutable latest container tags.
3. Namespace metadata matches the environment folder destination.

Verification and Execution Output

When promoting a hotfix or feature from staging to prod, the developer updates the production kustomization.yaml digest hash via PR. CI executes our test runner:

Once merged to main, ArgoCD’s root app evaluates the state across cluster target API endpoints. We verify sync states via the argocd CLI tool directly from our deployment pipeline:

To sync production safely, we execute a dry-run diff check before sending the final trigger:

Results and Measured Impact

After migrating all 24 microservices into this single-repo GitOps layout, we measured operational performance over a nine-month period:

• Deployment-induced Incidents: Dropped from 14 incidents per quarter to 0 across the last three consecutive quarters.
• Production Deployment Duration: Mean-time-to-deploy (MTTD) decreased from 45 minutes (manual cherry-picking, complex Helm overrides, cross-repo synchronization) to 4 minutes (single PR merge and dry-run validation).
• Configuration Drift: Zero instances of drift. ArgoCD flags and alerts on manual kubectl interventions within 180 seconds.
• Onboarding Overhead: New engineers onboard and ship their first dev-environment service deployment in under 2 hours without breaking downstream environments.

Lessons Learned

1. Ban Dynamic Floating Tags Everywhere: Relying on tags like :latest, :dev, or :staging in Kustomize introduces non-deterministic deployments where ArgoCD reports Synced even though underlying container byte outputs differ between node pulls. Use explicit SHA digests in production, enforced via CI python validation scripts.
2. Resist Shared Values Files: Early on, we attempted to create a base/common-config.env that all environments imported. This caused immediate coupling. If staging needs a new parameter that prod isn’t ready for yet, shared files break. Keep environment overlays completely decoupled from one another. Duplicate five lines of static key names if it avoids a global blast radius.
3. Explicitly Limit Auto-Sync Scope: Never enable selfHeal and automated syncing on production trading namespaces without human-in-the-loop diff confirmations. Automated sync belongs in dev and ephemeral branch environments; production requires manual triggers backed by automated pre-sync CI checks.