Prefix caching in vLLM: cutting TTFB for long system prompts

In high-throughput, LLM-powered trading intelligence pipelines, latency is the bottleneck that kills execution edge. My team runs an automated pipeline that ingests raw SEC filings, earnings transcripts, and alternative data feeds, processing them through a fine-grained, 8,500-token system prompt containing complex risk frameworks, financial taxonomy rules, and precise JSON schema definitions.

Using vanilla LLM generation setups, our Time to First Token (TTFB) hovered around 3.2 seconds on an 8x H100 GPU node. When you are processing hundreds of incoming filings concurrently, spending over three seconds just calculating the Key-Value (KV) cache for the same static system prompt on every single query is unacceptable waste.

To solve this, we integrated automatic prefix caching (APC) in vLLM. This optimization bypasses the prefill phase for shared prompt prefixes by retaining their KV cache in GPU memory across requests. Here is how we designed, implemented, and tuned our prefix caching system, the dead-ends we hit along the way, and the concrete code required to run it in production.

The Problem: Prefill Bottlenecks in Multi-Agent Pipelines

Every LLM inference request goes through two distinct execution phases:
1. Prefill: The engine processes the input prompt tokens, computes the keys and values for attention, and generates the first token. This is highly parallelizable but compute-bound.
2. Decoding: The engine generates subsequent tokens one by one. This is memory-bandwidth bound.

For an 8,500-token prompt, the prefill phase is incredibly heavy. Even with Tensor Parallelism (TP) spread across multiple GPUs, the engine spends billions of floating-point operations parsing those identical 8,500 tokens before it can emit a single character of output.

flowchart LR
 A["Client Request"] --> B["vLLM Router"]
 B --> C["Radix Cache Engine"]
 C -->|"Cache Hit"| D["Reused KV Blocks"]
 C -->|"Cache Miss"| E["Prefill Compute"]

Our initial architecture naively fed the entire system prompt along with the dynamic user prompt (the financial text to analyze) to the inference server. As traffic scaled during earnings season, our H100s were pinned at 100% compute utilization, not because we were generating millions of output tokens, but because we were re-calculating the exact same attention matrices for our risk instructions over and over again.

Dead End 1: Manual Prompt Chunking & Client-Side Caching

Before adopting vLLM’s native prefix caching, we tried splitting prompts on the client side, sending pre-tokenized chunks, and attempting to maintain session-state affinities across our GPU cluster. This was a disaster. Network serialization overhead swallowed any minor computational savings, and managing dynamic KV-cache eviction policies at the application level added thousands of lines of fragile state-synchronization code.

Dead End 2: Static Token Padding

We attempted to pad our system prompts to fixed token boundaries (e.g., aligning everything to exactly 8,192 tokens) using dummy spaces, thinking it would make cache keys deterministic across different LLM engine processes. However, tokenizer behavior is highly non-linear. Minor variations in whitespace sequences resulted in different token boundaries, completely invalidating the cache keys and degrading model coherence.

The Approach: Automatic Prefix Caching (APC) via Radix Attention

vLLM solves this natively using a data structure called a Radix Tree over the KV cache blocks. Instead of viewing the KV cache as a linear buffer associated with a single request, vLLM treats the cache as a tree where nodes represent sequences of tokens.

• When a request arrives, vLLM tokenizes the input and performs a prefix search on the Radix Tree.
• If a prefix match is found (e.g., the system prompt), the engine directly reuses the physical GPU memory blocks containing the KV cache for those tokens.
• The prefill phase is executed only for the remaining, unmatched suffix of the prompt (the dynamic user query).
• If a cache miss occurs, the prefix is processed normally, and its resulting KV cache blocks are inserted into the Radix Tree for future requests.
• When GPU memory runs low, vLLM evicts cached prefixes using a Least Recently Used (LRU) policy.

Crucially, this process is entirely transparent to the client. There is no need to pass special session IDs or manage block allocation manually.

The Code: Implementing and Benchmarking Prefix Caching

Below is the complete implementation of our high-performance benchmarking harness and production-ready vLLM configuration.

This code does three things:
1. Configures and spins up an in-process vLLM engine with prefix caching enabled.
2. Simulates our typical workload: a large, invariant system prompt (representing our financial analysis rules) coupled with unique, dynamic user queries.
3. Measures and compares the TTFB and generation speed of a “cold” request (cache miss) against subsequent “warm” requests (cache hits).

If you prefer running vLLM as an independent OpenAI-compatible API server—which is our preferred production setup to decouple the model deployment from our pipeline application logic—you can enable prefix caching via the CLI using the --enable-prefix-caching flag.

Here is the exact production-ready Bash command we use to launch the vLLM server:

To call this server asynchronously while tracking TTFB on the client side, we use this Python testing script:

Results: Before vs. After Prefix Caching

The performance improvement of enabling prefix caching for our high-token system prompts is dramatic. Below is the representative output of running the client-side streaming benchmark against our vLLM instance hosting the model on NVIDIA A100-SXM4-80GB GPUs:

Resource Utilization Profiles

The resource profiles paint a clear picture of why this occurs:

When prefix caching is active, the engine completely bypasses the prefill calculations for the cached tokens. It simply looks up the existing pointers in memory and feeds them straight to the attention block keys, resulting in instantaneous token emission.

Lessons: The Operational Realities of Prefix Caching

While prefix caching yielded immense speedups, getting it right in our production deployment required learning several harsh truths about vLLM’s memory manager.

1. Watch Out for Exact Tokenizer Alignment

Prefix caching is hash-based. If your client library modifies even a single character in your system prompt—such as appending an extra space, inserting a trailing newline, or dynamically swapping in a current timestamp—the hash will mismatch.

If your hash changes, you suffer a cold-prefill cache miss. We had to enforce strict linting and sanitization rules in our API gateway to strip whitespace and guarantee that our system prompts remained character-for-character identical across all client calls.

2. Block Allocation Size and Memory Fragmentation

vLLM manages GPU memory by splitting the KV cache into small virtual blocks (typically of size 16 tokens). If your system prompt is not a multiple of 16 tokens, the remaining trailing tokens will fill a partial block.

To maximize alignment and cache utilization efficiency, ensure your system prompts end precisely at block boundaries, or configure --block-size 16 (or 8) in your engine initialization script to minimize block fragmentation.

3. Eviction Pressure and Monitoring

Cached blocks are maintained using an LRU cache policy. Under intense load with multiple different system prompts, the engine will aggressively evict older prefixes to free up space for active decode operations.

We monitor this in production by scraping vLLM’s Prometheus metric endpoint:

If the gpu_cache_usage_factor continuously hits 1.0 while vllm:num_requests_waiting is rising, it indicates your GPU memory is saturated. You will need to either decrease gpu_memory_utilization to leave safety headroom, scale out to a larger tensor parallel size, or increase hardware count to prevent cache thrashing.