KV cache sizing for multi-GPU vLLM deployments

We run high-frequency, event-driven sentiment analysis pipelines that parse corporate earnings transcripts, SEC filings, and real-time news feeds. Our pipeline feeds trading signals directly to an execution router. In this world, p99 latency spikes are not just annoying—they are direct financial losses.

We migrated our primary inference pipeline from a custom Hugging Face transformers pipeline to vLLM running Llama-3-70B-Instruct across an 8-way NVIDIA A100 (80GB SXM4) node. vLLM’s PagedAttention is the gold standard for high-throughput LLM serving, but we immediately hit a wall. During periods of high market volatility, our system threw CUDA Out-Of-Memory (OOM) errors, suffered massive latency degradation, and dropped connection requests.

This is the post-mortem of how we debugged, measured, and solved the multi-GPU KV cache allocation problem on vLLM.

The Failure Mode: Live OOMs and Stuttering Engines

We initialised our vLLM engine with standard production settings. Since we were running a 70B parameter model in FP16, we split the model across 4 GPUs using Tensor Parallelism (tp_size=4), leaving the other 4 GPUs for a separate replica.

The configuration looked like this:

Under quiet market conditions (5 to 10 concurrent requests, average context length of 1,500 tokens), the server performed beautifully. Average time-to-first-token (TTFT) stayed under 40ms, and inter-token latency was stable at 12ms.

Then came an earnings-heavy Wednesday. Concurrency spiked to 85 concurrent requests. The context sizes ballooned as our pipeline began feeding complete 6,000-word financial reports into the context window.

Our logs lit up with this error:

The entire engine crashed. Because we were running in a Tensor Parallel group, when GPU 0 OOM’ed, ranks 1, 2, and 3 hung indefinitely waiting for a NCCL barrier synchronization, locking up the remaining healthy GPUs.

We had configured gpu_memory_utilization 0.90. This tells vLLM to reserve 90% of the total GPU memory (71.41 GB out of 79.35 GB) for the model weights and the KV cache. The remaining 10% (7.94 GB) was supposed to be a buffer for activation memory, workspace buffers, and temporary PyTorch allocations during execution.

But our math was wrong. We did not account for how Tensor Parallelism, Grouped Query Attention (GQA), and high concurrency interact under heavy load.

The Mechanics of KV Cache in Multi-GPU vLLM

To fix this, we have to look at how vLLM allocates memory at boot.

At startup, vLLM performs a profiling step. It loads the model weights across the specified tensor_parallel_size (TP). Then, it runs a dummy forward pass using the maximum possible input size to estimate the peak execution memory required for activations.

Whatever memory is left over within the gpu_memory_utilization budget is dedicated entirely to the KV cache block pool.

The Mathematics of Llama 3 70B KV Cache

Let us calculate the exact memory footprint of a single token’s KV cache for Llama 3 70B.

The model architecture parameters:
* Number of Layers ($L$): $80$
* Number of Key-Value Heads ($H_{kv}$): $8$ (It uses Grouped Query Attention; while query heads $H_q = 64$, the KV heads are grouped to $8$ to save memory)
* Head Dimension ($D_{head}$): $128$
* Bytes per Parameter ($B$): $2$ (FP16/BF16)

The formula for the KV cache size (in bytes) for a single token across the entire model:

$$\text{Memory per Token} = 2 \times L \times H_{kv} \times D_{head} \times B$$

The leading $2$ represents the fact that we must store both a Key vector and a Value vector.

Let’s plug in the numbers:

$$\text{Memory per Token} = 2 \times 80 \times 8 \times 128 \times 2 = 327,680 \text{ bytes} \approx 320 \text{ KB}$$

For a single sequence running at the maximum context window of $8,192$ tokens:

$$\text{Memory per Sequence} = 320 \text{ KB} \times 8192 = 2.62 \text{ GB}$$

If we have $TP = 4$, the model weights and the KV cache are sharded across the $4$ GPUs.
Each GPU hosts a subset of the KV heads. Since $H_{kv} = 8$ is divisible by $TP = 4$, each GPU processes exactly $8 / 4 = 2$ KV heads.

Thus, the KV cache footprint per token per GPU is:

$$\text{Memory per Token per GPU} = 2 \times 80 \times 2 \times 128 \times 2 = 81,920 \text{ bytes} \approx 80 \text{ KB}$$

For a maximum context sequence ($8,192$ tokens), each GPU must allocate:

$$\text{Memory per Sequence per GPU} = 80 \text{ KB} \times 8192 = 655.36 \text{ MB}$$

If we want to support $100$ concurrent sequences, our physical KV cache pool across the GPUs must hold:

$$\text{Total Cache Pool per GPU} = 100 \times 655.36 \text{ MB} = 65.54 \text{ GB}$$

The Memory Breakdown on a 80GB GPU

Let’s look at the actual allocation map of an 80GB A100 GPU when running Llama-3-70B with $TP=4$:

1. Model Weights: The 70B parameter model in FP16 takes up $140 \text{ GB}$ of physical space. Sharded over 4 GPUs:
   $$\text{Weights per GPU} = \frac{140 \text{ GB}}{4} = 35 \text{ GB}$$
2. PyTorch & CUDA Context Overhead: Approximately $1.5 \text{ GB}$ to $2.5 \text{ GB}$ of static driver overhead.
3. vLLM Peak Activation Memory: This is highly dependent on sequence length and batch size. During the profiling phase, vLLM allocates workspace buffers, NCCL communication ring buffers, and temporary tensor spaces. This typically scales up to $6 \text{ GB}$ to $8 \text{ GB}$ when max_model_len is set to $8,192$.
4. The Remaining Cache Space:
   $$\text{Available for Cache} = (80 \text{ GB} \times \text{utilization}) – \text{Weights} – \text{Overhead} – \text{Activations}$$

If we set gpu_memory_utilization = 0.90, we tell vLLM to claim $72 \text{ GB}$ of the GPU.
Subtracting $35 \text{ GB}$ for weights and $6 \text{ GB}$ for activation space leaves roughly $31 \text{ GB}$ for the local KV cache.

However, during high-concurrency bursts, PyTorch’s internal memory allocator (THCState) experiences fragmentation. When the scheduler attempts to execute an attention step with large batch sizes, it requests contiguous memory segments for activation tensors. If the remaining physical memory (the 10% “free” buffer, which is $8 \text{ GB}$) is fragmented or partially consumed by driver overhead, PyTorch throws a CUDA OOM.

Here is the data-flow inside vLLM when resolving these requests:

flowchart TD
 req["Request Stream"] --> sched["vLLM Scheduler"]
 sched --> manager["PagedAttention Block Manager"]
 manager -->|"Shard 0"| gpu0["GPU 0 KV Cache"]
 manager -->|"Shard 1"| gpu1["GPU 1 KV Cache"]
 gpu0 --> exec["Distributed Execution"]
 gpu1 --> exec

The Code: Profiling, Calculating, and Initializing Dynamically

To prevent these crashes, we built a wrapper script that inspects our current hardware topology, calculates the exact theoretical KV cache limits, and executes a dry-run calibration pass using vLLM’s internal API to determine the safest and most efficient parameters.

The script below executes this diagnostic and sets up the engine using the low-level LLMEngine API to guarantee predictable allocations.

When we executed this script against our hardware stack, the theoretical math matched vLLM’s internal profile output, exposing a hard boundary. Here is the CLI output from the run:

At gpu_memory_utilization=0.90, vLLM attempted to reserve too much physical memory for the blocks, causing PyTorch to crash immediately when initializing its communication handlers. Under 0.85, it booted cleanly and allocated $131,072$ blocks per GPU.

Production Deployment Optimization

Based on these profiling metrics, we moved away from the basic command-line configuration. We introduced a systematic orchestrator wrapper to spawn our production inference cluster.

We write our orchestrator config files with explicit attention to three parameters that control the scheduler’s behavior:

1. max_num_seqs: The maximum number of concurrent sequences the scheduler can execute in a single step. Keeping this in check limits the maximum transient activation memory.
2. max_model_len: Explicitly bound to $8,192$ to prevent long historical documents from driving up the block requirement exponentially.
3. gpu_memory_utilization: Hard-capped at 0.82 to preserve a safe $18\%$ buffer for the operating system and PyTorch runtime memory pools.

Here is our production deployment configuration manifest:

Results: System Behavior Under High Concurrency

After applying these constraints, we re-ran our load testing pipeline. This test mimics real-world market crashes: a massive burst of requests containing mixed historical chunks (large context windows) and rapid signal queries (short context).

When using the 0.90 default config, as concurrency rose above 30 requests, vLLM spent a significant amount of CPU cycles moving KV blocks back and forth between system RAM and GPU memory (block swapping). This caused inter-token latency to spike to 72ms.

With our calibrated config, the system rejected no requests, stopped swapping blocks, and avoided CUDA fragmentation. The engine scheduler stayed within its allocation limits, queuing requests smoothly rather than trying to process everything concurrently and crashing the underlying system.

Lessons Learned

1. Calculate, Don’t Guess: Do not blindly set your gpu_memory_utilization to 0.90 or 0.95 just because your model fits. Use the mathematical formula for GQA to calculate your footprint per token. Know your memory boundaries before you launch.
2. Tensor Parallelism Shrinks the Footprint Per-GPU: When using TP, both the model parameters and the KV heads are split. If your head count is not evenly divisible by your TP size, you will experience severe head-allocation imbalances across your GPUs.
3. Limit max_num_seqs: The scheduler needs activation space to execute a forward pass for all active sequences. If your sequence limit is set too high, the memory buffer required for activations will exceed your unallocated GPU capacity during concurrent bursts, leading to an immediate CUDA OOM.
4. Enforce Cooldowns and Cleansups: When running automated unit testing or live dynamic provisioning, always clean up CUDA references and force a garbage collection pass (gc.collect(), torch.cuda.empty_cache()). PyTorch will hold onto pointers, creating misleading OOM results during sequential boot profiling.