Speculative decoding on vLLM: the latency win and the gotchas
We run a real-time sentiment and market-impact parsing engine that processes news feeds, regulatory filings, and earnings transcripts. Our SLA requires us to parse, structure, and extract tradeable signals from incoming text blocks within 200 milliseconds.
Initially, we deployed a quantized Llama-3-70B model on an 8x H100 node using vLLM. While the throughput was excellent under heavy parallel load, our single-stream, auto-regressive generation latency was a bottleneck. Generating 150 tokens for a structured JSON payload took roughly 1.8 seconds—far too slow to act on fast-moving market feeds.
To solve this, we integrated speculative decoding into our production pipeline. Speculative decoding speeds up inference by running a smaller, faster “draft” model to generate candidate tokens, which are then verified in a single forward pass by the larger “target” model.
This post details the exact implementation, the performance gains we observed, and the structural gotchas that almost broke our production pipeline.
The Pipeline Architecture
The premise of speculative decoding is straightforward: target model generation is memory-bandwidth bound. Evaluating $N$ tokens in parallel takes nearly the same amount of time as evaluating a single token. By using a draft model (e.g., Llama-3-8B) to quickly generate a sequence of draft tokens, we can verify all of them in a single forward pass of our target model (Llama-3-70B).
flowchart LR PROMPT["User Prompt"] --> DRAFT["Draft Model Runs K Steps"] DRAFT -->|"Draft Tokens"| TARGET["Target Model Single Forward Pass"] TARGET -->|"Verify Logits"| DECIDE["Accept or Reject Decision"] DECIDE -->|"Accept"| UPDATE["Append Accepted Tokens to KV Cache"] DECIDE -->|"Reject"| ROLLBACK["Discard Rejected and Sample New"] UPDATE --> DRAFT ROLLBACK --> DRAFT
If the target model accepts the draft tokens, we get multiple tokens for the computational cost of one target model forward pass. If it rejects them, we fall back to standard auto-regressive generation for that step.
The Code: Implementation and Benchmarking
We set up a benchmarking suite to isolate the impact of speculative decoding under varying configurations.
Below is the shell script we used to spin up the speculative decoding server using vLLM, followed by our asynchronous benchmarking client.
1. Launching the Speculative vLLM Engine
To run this, you need a GPU configuration that can hold both models. We allocated 8 GPUs, using tensor parallelism (tp=8) for the 70B target model while running the 8B draft model alongside it.
# File: launch_vllm_speculative.sh
# Set up paths to your Hugging Face or local weights
TARGET_MODEL="meta-llama/Meta-Llama-3-70B-Instruct"
DRAFT_MODEL="meta-llama/Meta-Llama-3-8B-Instruct"
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
# Launch vLLM with speculative decoding configuration
python3 -m vllm.entrypoints.openai.api_server \
–model "$TARGET_MODEL" \
–tensor-parallel-size 8 \
–port 8000 \
–speculative-model "$DRAFT_MODEL" \
–num-speculative-tokens 5 \
–gpu-memory-utilization 0.90 \
–max-model-len 4096 \
–trust-remote-code \
–enforce-eager
2. The Async Benchmark Client
The benchmarking script below sends concurrent requests to our vLLM instance. It simulates our production structured JSON extraction task, tracking Time to First Token (TTFT), Inter-Token Latency (ITL), and total generation latency.
import asyncio
import time
import argparse
import json
import httpx
import numpy as np
PROMPT_TEMPLATE = """You are an algorithmic trading parser. Extract the ticker, sentiment score (-1.0 to 1.0), and target price from this news flash. Return valid JSON only.
News Flash:
"Acme Corp (ACME) announces Q3 revenue up 14% year-over-year, beating analyst expectations by $40M. CEO forecasts a strong Q4 and raises target guidance to $145.00."
JSON output:"""
async def send_request(client: httpx.AsyncClient, url: str, request_payload: dict) -> dict:
start_time = time.perf_counter()
ttft = 0.0
tokens_generated = 0
text_buffer = ""
try:
async with client.stream("POST", url, json=request_payload, timeout=60.0) as response:
if response.status_code != 200:
print(f"Error: {response.status_code}")
return {"status": "error"}
async for line in response.aiter_lines():
if not line.strip():
continue
if line.startswith("data: "):
data_str = line[6:]
if data_str.strip() == "[DONE]":
break
data = json.loads(data_str)
if not data["choices"]:
continue
delta = data["choices"][0]["delta"]
if "content" in delta:
if ttft == 0.0:
ttft = time.perf_counter() – start_time
text_buffer += delta["content"]
tokens_generated += 1
total_time = time.perf_counter() – start_time
return {
"status": "success",
"ttft": ttft,
"total_time": total_time,
"tokens_generated": tokens_generated,
"itl": (total_time – ttft) / max(tokens_generated – 1, 1),
"text": text_buffer
}
except Exception as e:
print(f"Exception during request: {str(e)}")
return {"status": "exception"}
async def run_benchmark(host: str, port: int, concurrency: int, num_requests: int):
url = f"http://{host}:{port}/v1/chat/completions"
payload = {
"model": "meta-llama/Meta-Llama-3-70B-Instruct",
"messages": [
{"role": "user", "content": PROMPT_TEMPLATE}
],
"temperature": 0.0, # Highly deterministic for structured extraction
"max_tokens": 150,
"stream": True
}
limits = httpx.Limits(max_keepalive_connections=concurrency, max_connections=concurrency)
async with httpx.AsyncClient(limits=limits) as client:
# Warmup request
print("Running warmup request…")
await send_request(client, url, payload)
print(f"Starting benchmark: Concurrency={concurrency}, Total Requests={num_requests}")
start_wall_clock = time.perf_counter()
tasks = []
# Simple queue to manage concurrency
sem = asyncio.Semaphore(concurrency)
async def worker():
async with sem:
return await send_request(client, url, payload)
for _ in range(num_requests):
tasks.append(worker())
results = await asyncio.gather(*tasks)
wall_clock_time = time.perf_counter() – start_wall_clock
# Filter successful runs
valid_results = [r for r in results if r.get("status") == "success"]
if not valid_results:
print("No successful runs completed.")
return
ttfts = [r["ttft"] * 1000 for r in valid_results] # in ms
total_times = [r["total_time"] * 1000 for r in valid_results] # in ms
itls = [r["itl"] * 1000 for r in valid_results] # in ms
tokens = [r["tokens_generated"] for r in valid_results]
print("\n=== BENCHMARK RESULTS ===")
print(f"Total Wall Clock Time: {wall_clock_time:.2f} s")
print(f"Successful Requests: {len(valid_results)}/{num_requests}")
print(f"Throughput: {len(valid_results) / wall_clock_time:.2f} req/sec")
print(f"Total Tokens Generated: {sum(tokens)}")
print(f"Tokens/Sec (Aggregate): {sum(tokens) / wall_clock_time:.2f}")
print(f"Mean TTFT: {np.mean(ttfts):.2f} ms (p95: {np.percentile(ttfts, 95):.2f} ms)")
print(f"Mean ITL (Inter-Token Latency): {np.mean(itls):.2f} ms (p95: {np.percentile(itls, 95):.2f} ms)")
print(f"Mean Execution Time: {np.mean(total_times):.2f} ms (p95: {np.percentile(total_times, 95):.2f} ms)")
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.parser_output = parser.add_argument("–host", type=str, default="localhost")
parser.add_argument("–port", type=int, default=8000)
parser.add_argument("–concurrency", type=int, default=4)
parser.add_argument("–requests", type=int, default=20)
args = parser.parse_args()
asyncio.run(run_benchmark(args.host, args.port, args.concurrency, args.requests))
Performance Results
We ran the benchmarks on identical hardware configurations (8x H100 SXM, Tensor Parallelism 8) with and without speculative decoding. The results below showcase the impact of using Llama-3-8B as a speculative draft model for Llama-3-70B on our highly predictable structured parser output.
| Metric | Target Only (Llama-3-70B) | Speculative Enabled (Llama-3-70B + Llama-3-8B) | Improvement Factor |
|---|---|---|---|
| Mean Inter-Token Latency (ITL) | 12.8 ms | 5.2 ms | 2.46x Faster |
| Mean Execution Time (150 Tokens) | 1,920 ms | 780 ms | 2.46x Faster |
| p95 Execution Time | 2,150 ms | 890 ms | 2.41x Faster |
| Mean Time to First Token (TTFT) | 18.5 ms | 23.1 ms | 24% Slower |
| GPU Memory Overhead | ~72% VRAM | ~89% VRAM | Higher Overhead |
When analyzing vLLM’s internal metrics via the console logs during execution, we observed the following acceptance rates:
INFO 10-24 14:32:01 metrics.py:345] Draft acceptance rate: 76.4%
INFO 10-24 14:32:01 metrics.py:345] System efficiency gain: 1.74x
INFO 10-24 14:32:01 metrics.py:345] Avg verified tokens per step: 3.82 / 5.00
The Trade-off
The 2.4x drop in inter-token latency is excellent, but it comes with two primary costs:
1. TTFT Overhead: Time to First Token increased by roughly 5 ms. This is because the initial draft token must be sampled, processed, and validated before the first token is flushed.
2. VRAM Footprint: Running the 8B model alongside the 70B model requires allocating memory for the draft model’s weights and, crucially, its own KV cache.
Production Gotchas and Dead-ends
Our transition to speculative decoding in production was not entirely smooth. Below are the key issues we encountered.
1. The Tokenizer Mismatch Trap
Initially, we attempted to use TinyLlama-1.1B as a lightweight draft model for our Llama-3-70B engine to conserve VRAM. The system crashed on startup with a tokenizer mismatch error.
Speculative decoding in vLLM requires both the target model and the draft model to use the exact same vocabulary and tokenizer. If the token IDs do not map to the identical string representations, the verification pass will continuously reject tokens, collapsing the draft acceptance rate to near-zero or crashing the engine.
For Llama-3-70B, you must use a model that shares the same tokenizer family, such as Llama-3-8B. If you require a smaller draft model, you must use one specifically distilled or pruned from the exact same parent vocabulary.
2. High Temperature Collapses Acceptance Rates
Our evaluation system initially ran creative writing and open-ended summary pipelines alongside our trading extraction pipelines. On tasks with high temperatures (temperature >= 0.8), we noticed speculative decoding throughput fell below baseline performance.
With high temperature or highly creative generation tasks, the draft model and target model probabilities diverge significantly. The target model rejects the proposed tokens, leading to wasted steps:
INFO 10-24 14:45:10 metrics.py:345] Draft acceptance rate: 12.1%
When acceptance rates fall below ~20%, speculative decoding becomes slower than standard execution. You are paying the latency price of running the draft model forward pass and the target model verification pass, only to discard the draft tokens and run standard sampling anyway.
3. KV Cache Allocation and Out-of-Memory (OOM) Errors
vLLM utilizes PagedAttention to manage KV caches. Under high load, we began hitting CUDA Out-of-Memory exceptions.
Adding the draft model forces vLLM to split its available GPU memory between:
– Target model weights
– Target model KV cache
– Draft model weights
– Draft model KV cache
If you do not tune gpu-memory-utilization and limit concurrent requests, the system will run out of overhead memory during peak concurrent traffic. To prevent this, we had to reduce our target model’s gpu-memory-utilization to 0.85 and lower the maximum context length (max-model-len) to ensure stability under load.