Monitoring GPU memory fragmentation in long-running inference servers

Six months ago, our team was running a cluster of 64 NVIDIA H100 GPUs hosting a pipeline of custom Stable Diffusion XL and LLaMA-3-70B models. For the first 36 hours of deployment, our systems ran smoothly, maintaining a p99 latency of 85ms for image generation and 22ms per token for text generation.

Then, at 3:00 AM on a Tuesday, three nodes crashed simultaneously with the infamous torch.cuda.OutOfMemoryError: CUDA out of memory.

Our Grafana dashboards, which were pulling standard metrics from nvidia-container-toolkit and Prometheus, showed that the GPU memory usage on those nodes had been flat at 82% for twelve hours leading up to the crash. There were no sudden spikes in batch size, no memory leaks in our Python process heap, and no changes in model weights.

We had fallen victim to GPU memory fragmentation—the silent killer of long-running inference servers.

This post walks through the mechanics of why this happens, why standard monitoring tools fail to catch it, and how to build a robust, real-time observability pipeline to detect and mitigate fragmentation before it takes down your production servers.

Why standard observability tools fail

Most machine learning infrastructure teams rely on nvidia-smi or the Prometheus Exported dcgm-exporter to monitor GPU memory. This is a critical mistake for long-running processes.

nvidia-smi queries the NVIDIA driver via NVML (NVIDIA Management Library). It reports physical memory allocation: how much virtual memory space the GPU driver has mapped to a particular process ID.

Output:

According to NVML, we have nearly 9.4 GB of free memory. But when our PyTorch inference loop attempts to allocate a contiguous block of 4 GB for a dynamic batch of image generations, it crashes:

Why does this happen? The answer lies in the PyTorch Caching Allocator.

To avoid the massive overhead of calling the CUDA driver API (cudaMalloc and cudaFree) on every single forward pass, PyTorch manages its own memory arena. When a tensor is freed in Python, PyTorch doesn’t return that memory to the operating system or the GPU driver. It keeps the block in an internal pool of idle memory so it can be instantly reused for subsequent allocations.

Over days of continuous operations with dynamic input sizes (variable token lengths, varying image dimensions, dynamic batching), this pool becomes highly fragmented. PyTorch’s allocator splits larger memory blocks into smaller segments to satisfy small allocations. Over time, these segments become interleaved.

When a request arrives requiring a large contiguous chunk of memory, the allocator cannot find a single block big enough, even though the sum of all free space in the pool is far larger than the requested size.

flowchart TD
 client["Client App"] -->|"/v1/chat/completions"| proxy["Load Balancer"]
 proxy -->|Route Request| infServer["Inference Engine (FastAPI/vLLM)"]
 infServer -->|"PyTorch Allocator Requests"| gpu["GPU H100"]
 metrics["Observability Exporter (Python)"] -->|"Scrapes torch.cuda Stats"| infServer
 metrics -->|"Exposes /metrics"| prometheus["Prometheus"]
 prometheus -->|"Triggers Webhook"| alertManager["Alertmanager"]

To diagnose this issue, we must query the PyTorch allocator directly using its internal diagnostics APIs.

Exploring the PyTorch Caching Allocator internals

PyTorch exposes raw allocator statistics through torch.cuda.memory_stats() and torch.cuda.memory_snapshot(). To build our monitoring solution, we must understand the core metrics returned by these APIs.

The most critical metrics for detecting fragmentation are:

1. allocated_bytes: The amount of memory currently held by active tensor objects.
2. reserved_bytes: The total memory currently held in PyTorch’s caching allocator pool (this corresponds to what the GPU driver/NVML sees as “used” by the process).
3. inactive_split_bytes: The amount of memory held in blocks that are currently inactive but cannot be released back to the driver or reused for larger allocations because they are trapped within larger split blocks. This is your primary indicator of fragmentation.

Let’s write a small Python script to inspect these metrics in real-time under a simulated dynamic workload.

Output of our simulation:

Here, PyTorch has reserved 70 MB of physical GPU memory, but only 36 MB is actively holding tensors. 24 MB is locked up as “inactive split” memory. This means approximately 34% of our reserved memory is dead weight, unusable for larger, contiguous allocations.

Building a production-grade Prometheus metrics exporter

To monitor this across hundreds of GPUs in a production cluster, we need to export these internal PyTorch metrics to Prometheus.

The following code implements a clean, production-grade metrics exporter using the Prometheus Python client. It exposes custom gauges that parse the deep nested dictionary structure of torch.cuda.memory_stats() and exposes a clean metrics endpoint.

To expose this inside an active inference server like a FastAPI app (which acts as our LLM proxy/orchestrator), we can spin up this exporter in a background thread or embed the metric registry directly into the server application.

Here is an integration using a background thread within a FastAPI application shell:

Querying and alert rules in Prometheus

With the metrics flowing to our Prometheus instance, we can configure dynamic monitoring and alerts.

The metrics scraped from our endpoint expose the precise state of the caching allocator. Here is a sample scrape payload returned by our exporter:

We configure Prometheus alerts inside Alertmanager. Rather than alerting when physical memory is full (which is normal for vLLM or PyTorch servers that pre-allocate KV caches), we alert when the fragmentation ratio is persistently high AND allocation retries are climbing.

This is the rule file we deployed to production:

Real-world results and dynamic mitigation

Before implementing this observability system, we had no warning system for our crashes. The standard metric container_gpu_memory_used_bytes remained at a rock-solid 90% until the moment the node completely went down:

After deploying the exporter, our observability dashboard revealed what actually happened.

The fragmentation ratio climbed monotonically over 14 hours from 0.08 to 0.42. During this entire time, NVML reported a flat 90% memory consumption. The moment the ratio crossed 0.40, a prompt with a slightly longer context sequence came in, forcing a request for a 3.5 GB contiguous block. The allocator failed, retried, and crashed the engine.

With these custom metrics, we built a simple, automated self-healing loop:

1. Detection: If gpu_pytorch_fragmentation_ratio exceeds 0.35 for over 5 minutes.
2. Mitigation:
3. Mark the affected container as Unhealthy on our Kubernetes load-balancer. This gracefully drains incoming HTTP/gRPC requests away from the node.
4. Wait 30 seconds for active requests to complete.
5. Call torch.cuda.empty_cache() inside the server thread when the request queue reaches zero.
6. If the fragmentation ratio remains above 0.20 even after clearing the cache, trigger a rolling pod restart.

By implementing this proactive, metric-driven draining mechanism, our service availability improved from 99.2% to 99.99%, saving our team from 3 AM on-call incidents.

Hard-won architectural lessons

1. torch.cuda.empty_cache() is not a silver bullet: Do not run this on every inference request. Running empty_cache() releases all unused cached memory back to the GPU driver. However, this operation causes a full GPU synchronization block. This stalls your execution pipelines and introduces latency spikes from 20ms to upwards of 1200ms for incoming requests. Use it only when the server is idle or during active draining.
2. Set PYTORCH_CUDA_ALLOC_CONF wisely: If you are dealing with diverse, dynamically sized inputs, tune your caching allocator configuration. Exporting the following environment variable before starting your Python runtime instructs PyTorch to use a round-up allocation strategy, grouping allocations into power-of-two size blocks, which dramatically reduces internal fragmentation:
   bash
export PYTORCH_CUDA_ALLOC_CONF="max_split_size_mb:256,roundup_power2_divisions:2"
3. Monitor at the Engine Level: Do not rely on system agents running outside your container namespace to tell you why your AI workload is crashing. Build your monitoring inside the runtimes that manage the execution context.