ArgoCD app-of-apps: structuring GitOps for a dozen microservices

Eighteen months ago, my team transitioned our core algorithmic execution engine from a tightly coupled Django monolith to twelve specialized microservices. We run a mix of high-frequency order routers, real-time risk evaluation engines, data-ingestion pipelines, and historical reporting systems.

When we had two services, managing them with raw Kubernetes manifests and some manual Helm charts was fine. But as we scaled to a dozen services across three distinct environments (development, staging, and production), deployment operations collapsed under their own weight. We suffered from configuration drift, silent deployment failures, and a painful “chicken-and-egg” bootstrapping process when spin-up times for new developer environments stretched to hours.

We turned to GitOps and ArgoCD to establish a single source of truth. However, managing twelve distinct Application manifests manually inside ArgoCD quickly became a secondary operational nightmare. Every time we added a new service, we had to manually apply a new YAML definition to the management cluster or run ad-hoc argocd app create commands.

The breakthrough came when we adopted the app-of-apps pattern. This post details how we structured this pattern, the concrete code we use in production today, the major failures we encountered along the way, and the exact architectural patterns that solved them.

The Architecture: App-of-Apps

The core concept of the app-of-apps pattern is to treat your infrastructure and application definitions recursively. Instead of asking ArgoCD to track a dozen separate applications individually, you point ArgoCD to a single “Root” Application.

This Root Application does not contain services like your database or your billing API directly. Instead, it contains templates that generate other ArgoCD Application custom resources. When the Root Application syncs, it applies these child Application manifests to the cluster. ArgoCD then detects these new child definitions and automatically provisions the actual workloads (deployments, services, ingress) they point to.

This creates a clean, hierarchical dependency tree that we can manage entirely within a single Git repository.

flowchart TD
 git["Git Repository"]
 root["Root Application"]
 argocd["ArgoCD Controller"]
 appA["Service A Application"]
 appB["Service B Application"]
 k8s["Kubernetes Cluster Engine"]
 git -->|"Polls manifest changes"| argocd
 argocd -->|"Deploys root"| root
 root -->|"Spawns child definitions"| appA
 root -->|"Spawns child definitions"| appB
 appA -->|"Applies resources"| k8s
 appB -->|"Applies resources"| k8s

Directory Layout of the GitOps Repository

To make this scale across dev, staging, and production without massive duplication, we separated our GitOps repository into a structured layout using Helm. We chose Helm as the templating engine for our app-of-apps configuration because it allows us to loop through service definitions dynamically using simple configuration blocks (values.yaml).

This is the exact directory layout we use:

• bootstrap/root-app.yaml: The bootstrap manifest that you apply once to your cluster to kick off the entire process.
• apps/: The Helm chart representing our app-of-apps. It iterates over configuration values to generate ArgoCD Application manifests.
• services/: The actual Helm charts or Kustomize templates for our underlying microservices. For maximum decoupling, these can also reside in external application repositories; in our case, our CI pipelines build the Docker images and push updated tags to values files inside this central directory.

The Code: Implementing the App-of-Apps Pattern

Let us look at the code required to make this work. We start with the Root Application definition that bootstraps the entire setup.

1. The Bootstrap Manifest (bootstrap/root-app.yaml)

This manifest is applied manually once using kubectl apply -f bootstrap/root-app.yaml. It configures ArgoCD to look at the apps/ directory in our repository and track it.

2. The App-of-Apps Chart Metadata (apps/Chart.yaml)

Inside the apps/ directory, we define a standard Helm chart that does nothing other than template ArgoCD Application manifests.

3. The Central Templating Engine (apps/templates/applications.yaml)

This template is the heart of the app-of-apps pattern. It loops through a dictionary of microservices defined in our values file and generates an ArgoCD Application CRD for each one.

4. Environment-Specific Configurations (apps/values-prod.yaml)

By passing different value files to the root application, we can control which services are active in which environments, which sync waves they belong to, and what Git branches or tags to track.

Two Critical Failures (And How We Resolved Them)

We did not arrive at this configuration smoothly. Two major operational failures almost made us scrap the entire migration and return to running script-based deployment pipelines.

Failure 1: The Cascading Delete Apocalypse

Early on, we did not understand the true power—and danger—of the ArgoCD finalizer metadata block:

During a routine cleanup of a legacy development namespace, a senior engineer decided to rebuild the Root Application. They ran:

Because the Root Application generated all child applications, and those child applications contained the resources-finalizer.argocd.argoproj.io finalizer, Kubernetes executed a cascading delete.

ArgoCD did exactly what it was programmed to do: it began recursively deleting every single child application, which in turn wiped out every single deployment, service, PersistentVolumeClaim, and database instance across the entire cluster in less than three minutes. Our staging cluster was completely flattened.

The Fix

1. Remove finalizers from the Root Application: We stripped the finalizer from the root-app.yaml itself. This ensures that if the root application is deleted, it merely deletes the definitions of the child apps in ArgoCD, without triggering a destructive, cascading purge of the resources inside the cluster.
2. Selective cascading deletes: We only append finalizers to stateless workloads where a clean teardown is desired. For databases or stateful components, we omit the finalizer or use non-cascading deletion policies:

Failure 2: The Infinite Sync Loop (Git Write-Back Drift)

We configured our CI pipeline (GitHub Actions) to automatically update the Docker image tags inside our services/*/values.yaml files on every successful merge to the main branch.

However, we also had a post-deployment database migration job in Kubernetes that, upon running successfully, updated a custom configmap in the cluster containing schema state metadata.

This setup created an infinite reconciliation loop:

1. CI pushed an updated image tag to values.yaml in Git.
2. ArgoCD detected the commit, synced the application, and deployed the migration job.
3. The migration job updated the schema status configmap in the cluster.
4. Because our Git repository lacked this run-time status update, ArgoCD marked the application as OutOfSync (drift detected between live state and desired Git state).
5. ArgoCD’s selfHeal kicked in, rolled back the configmap changes, triggered the migration job again, which updated the configmap again, restarting the entire cycle.

Our logs were filled with hundreds of reconciliation events, creating excessive API load on the Kubernetes control plane and running up our cloud compute bill.

The Fix

We modified our child application templates using the ignoreDifferences block. This instructs the ArgoCD engine to explicitly ignore changes made by live controllers or run-time jobs to specific fields in our cluster:

This immediately broke the feedback loop. ArgoCD now tracks configuration changes in Git, but ignores the necessary run-time updates written by our database schema migration tools.

Results and Metrics

Adopting the app-of-apps pattern yielded immediate, quantifiable improvements across our deployment pipeline.

Operational Lessons

If you are planning to transition multiple microservices to an app-of-apps pattern, keep these operational rules in mind:

1. Leverage Sync Waves Aggressively: Do not let your applications spin up in a random race condition. Use argocd.argoproj.io/sync-wave annotations to ensure your service-mesh, operators, and databases are healthy and running before your API pods try to connect to them.
2. Keep the Root Simple: Resist the temptation to put common configmaps or secrets directly in the Root Application chart. Keep the root purely as an orchestrator of other applications.
3. Set Up Notifications Early: Use the ArgoCD Notification controller to send alerts directly to Slack or Teams when the Root Application enters a degraded state. If the root app fails, your entire deployment pipeline is blind to downstream changes.