Building a self-hosted LLM inference stack on a budget
We were spending $4,200 a month on closed-source LLM APIs to power a real-time sentiment extraction and limit-order-book feature pipeline. The setup was simple but brittle: every minute, our workers ingested market news, parsed structural shifts, and hit external endpoints.
Then we hit the scaling wall. During high-volatility events, our API costs spiked exponentially, and rate-limiting throttled our trading signals exactly when we needed them most. Even worse, the round-trip latency to external endpoints fluctuated from 250 milliseconds to over 4 seconds. For systematic trading, this variability is a death sentence.
I decided to pull everything in-house. This is the blueprint of how we designed, built, and optimized a production-grade, self-hosted LLM inference stack using consumer-grade budget GPU hardware, saving us thousands of dollars a month while dropping our p99 latency to a predictable 120ms.
The Hardware Bottleneck: Choosing a Budget GPU
When you build a self-hosted LLM inference stack on a budget, your biggest constraint is VRAM. If your model’s weights do not fit entirely onto the GPU’s onboard memory, you slide back into system RAM fallback, and your generation speeds drop from 50 tokens per second to a useless 1.5 tokens per second.
We looked at cloud GPUs first. An on-demand NVIDIA A100 (80GB VRAM) costs roughly $3.50 per hour on major clouds. Over a year, that is over $30,000 in operational expenditure per instance—hardly a budget play.
Instead, we chose the secondary market king: the NVIDIA RTX 3090 (24GB VRAM). You can easily buy used RTX 3090s for around $650 to $700. Two of these cards give you 48GB of aggregate VRAM for under $1,500.
The PCIe Lane Trap
In our first build, we threw two RTX 3090s into a consumer-grade motherboard running an Intel Core i7 with only 16 PCIe CPU lanes. We configured vLLM to run Tensor Parallelism (tp_size=2) to split our model across both cards.
It performed terribly. Every attention layer update required rapid syncs between the two GPUs. Because our motherboard split the PCIe lanes into x4/x4 configurations when both slots were populated, the inter-GPU communication bottlenecked the system.
To solve this without spending $5,000 on an enterprise AMD EPYC workstation, we sourced a used AMD Threadripper 1920X platform with 64 PCIe lanes. This allowed us to run both GPUs at full PCIe Gen 3 x16 speeds.
The Software Stack Architecture
To achieve high throughput, we bypassed basic Hugging Face pipelines or naive llama.cpp implementations. We went with vLLM as our execution engine. vLLM uses PagedAttention, which radically reduces memory fragmentation by managing key-value (KV) caches in page-allocated tables, much like virtual memory in operating systems.
Our architecture routes external client requests through a FastAPI gateway, which handles input validation, token rate-limiting, and metric logging. FastAPI offloads the inference work to an asynchronous vLLM engine run via an internal Ray cluster configured across our two local GPUs.
flowchart TD Client["Client App"] -->|"HTTP/gRPC"| Gateway["FastAPI Gateway"] Gateway -->|"Async Queue"| Engine["vLLM Engine"] Engine -->|"TP=2 via Ray"| GPU0["RTX 3090 (GPU 0)"] Engine -->|"TP=2 via Ray"| GPU1["RTX 3090 (GPU 1)"] GPU0 <-->|"PCIe Peer-to-Peer"| GPU1
The Code: Step-by-Step Implementation
This is our production setup. It includes a Docker environment to pin our CUDA dependencies, our core FastAPI inference server powered by the asynchronous vLLM engine, and an automated benchmark script to stress-test your local setup.
1. The Environment Setup (Dockerfile)
CUDA version mismatches are a common failure point when combining vLLM, PyTorch, and Ray. This Dockerfile locks down our runtime environment.
FROM nvidia/cuda:12.1.1-devel-ubuntu22.04
ENV DEBIAN_FRONTEND=noninteractive
ENV PYTHONUNBUFFERED=1
RUN apt-get update && apt-get install -y \
python3-pip \
python3-dev \
git \
curl \
&& rm -rf /var/lib/apt/lists/*
RUN ln -s /usr/bin/python3 /usr/bin/python
# Install PyTorch compiled for CUDA 12.1
RUN pip install –no-cache-dir torch torchvision torchaudio –index-url https://download.pytorch.org/whl/cu121
# Install vLLM and production dependencies
RUN pip install –no-cache-dir \
vllm==0.3.3 \
fastapi==0.110.0 \
uvicorn==0.28.0 \
pydantic==2.6.4 \
prometheus-client==0.20.0 \
ray==2.10.0
WORKDIR /app
COPY . /app
EXPOSE 8000
ENTRYPOINT ["python", "server.py"]
2. The High-Performance Inference Server (server.py)
This server sets up the async vLLM engine to run Llama-3-8B-Instruct. If you are running dual RTX 3090s, you can set tensor_parallel_size=2 to pool the memory of both cards.
import os
import asyncio
from fastapi import FastAPI, HTTPException
from fastapi.responses import StreamingResponse
from pydantic import BaseModel, Field
from typing import List, Optional, AsyncGenerator
from vllm.engine.arg_utils import AsyncEngineArgs
from vllm.engine.async_llm_engine import AsyncLLMEngine
from vllm.sampling_params import SamplingParams
from vllm.utils import random_uuid
app = FastAPI(title="Budget LLM Inference Server")
# Configure Engine Settings
# Using Llama-3-8B-Instruct quantized to AWQ (4-bit) to maximize context window and speed
MODEL_PATH = "solidrust/Meta-Llama-3-8B-Instruct-AWQ"
engine_args = AsyncEngineArgs(
model=MODEL_PATH,
tensor_parallel_size=int(os.getenv("TENSOR_PARALLEL_SIZE", "2")),
gpu_memory_utilization=0.90, # Reserve 10% VRAM for overhead/workspace
max_model_len=8192,
quantization="awq",
trust_remote_code=True,
disable_log_stats=False # Useful for keeping track of batching efficiency
)
# Global engine reference
engine = AsyncLLMEngine.from_engine_args(engine_args)
class GenerateRequest(BaseModel):
prompt: str
temperature: Optional[float] = Field(default=0.2, ge=0.0, le=2.0)
top_p: Optional[float] = Field(default=0.9, ge=0.0, le=1.0)
max_tokens: Optional[int] = Field(default=256, ge=1, le=4096)
stop: Optional[List[str]] = Field(default=None)
class TokenPayload(BaseModel):
text: str
async def stream_results(results_generator) -> AsyncGenerator[str, None]:
async for request_output in results_generator:
# Yield only the newly generated text delta
text_outputs = request_output.outputs[0].text
yield text_outputs
@app.post("/v1/generate")
async def generate(request_data: GenerateRequest):
"""Non-streaming inference endpoint."""
request_id = random_uuid()
sampling_params = SamplingParams(
temperature=request_data.temperature,
top_p=request_data.top_p,
max_tokens=request_data.max_tokens,
stop=request_data.stop
)
try:
results_generator = engine.generate(
request_data.prompt,
sampling_params,
request_id
)
final_output = None
async for request_output in results_generator:
final_output = request_output
if final_output is None:
raise HTTPException(status_code=500, detail="No output generated.")
return {
"id": request_id,
"prompt": request_data.prompt,
"text": final_output.outputs[0].text,
"prompt_tokens": len(final_output.prompt_token_ids),
"completion_tokens": len(final_output.outputs[0].token_ids)
}
except Exception as e:
raise HTTPException(status_code=500, detail=str(e))
@app.post("/v1/generate_stream")
async def generate_stream(request_data: GenerateRequest):
"""Streaming inference endpoint for low TTFT (Time to First Token) requirements."""
request_id = random_uuid()
sampling_params = SamplingParams(
temperature=request_data.temperature,
top_p=request_data.top_p,
max_tokens=request_data.max_tokens,
stop=request_data.stop
)
try:
results_generator = engine.generate(
request_data.prompt,
sampling_params,
request_id
)
return StreamingResponse(
stream_results(results_generator),
media_type="text/event-stream"
)
except Exception as e:
raise HTTPException(status_code=500, detail=str(e))
@app.get("/health")
async def health():
"""Liveness probe to ensure the engine is loaded."""
return {"status": "healthy", "model": MODEL_PATH}
if __name__ == "__main__":
import uvicorn
uvicorn.run(app, host="0.0.0.0", port=8000, log_level="info")
3. The Orchestration Layer (docker-compose.yml)
This configuration maps our host GPUs into the Docker container, allows them to share memory segments for fast peer-to-peer data transfers, and unlocks memory pinning.
version: '3.8'
services:
inference-server:
build: .
container_name: llm_inference_stack
ports:
– "8000:8000"
environment:
– CUDA_VISIBLE_DEVICES=0,1
– TENSOR_PARALLEL_SIZE=2
volumes:
– ~/.cache/huggingface:/root/.cache/huggingface
deploy:
resources:
reservations:
devices:
– driver: nvidia
count: all
capabilities: [gpu]
ipc: host # Critical for fast Ray/vLLM shared memory communication
shm_size: '16gb' # Necessary to prevent out-of-memory issues with high-throughput batching
restart: always
4. The Benchmarking Client (benchmark.py)
Do not rely on toy UI interactions to measure performance. Use this asynchronous client simulation to hit your new server with hundreds of concurrent API calls. This allows you to verify that your budget setup can handle high concurrency under load.
import asyncio
import aiohttp
import time
import statistics
API_URL = "http://localhost:8000/v1/generate"
CONCURRENT_REQUESTS = 20
TEST_PROMPT = "Explain the difference between limit orders and market orders in high frequency trading in two detailed paragraphs."
async def send_request(session: aiohttp.ClientSession, payload: dict) -> float:
start_time = time.perf_counter()
try:
async with session.post(API_URL, json=payload) as response:
if response.status == 200:
result = await response.json()
latency = time.perf_counter() – start_time
tokens = result["completion_tokens"]
tps = tokens / latency
return latency, tps, True
else:
return 0.0, 0.0, False
except Exception:
return 0.0, 0.0, False
async def main():
payload = {
"prompt": TEST_PROMPT,
"temperature": 0.1,
"max_tokens": 150
}
print(f"Warm up: sending test request to {API_URL}…")
async with aiohttp.ClientSession() as session:
# Warm-up request to compile kernels on GPU
await send_request(session, payload)
print(f"Starting benchmark with {CONCURRENT_REQUESTS} parallel clients…")
start_benchmark = time.perf_counter()
tasks = [send_request(session, payload) for _ in range(CONCURRENT_REQUESTS)]
results = await asyncio.gather(*tasks)
total_benchmark_time = time.perf_counter() – start_benchmark
latencies = [r[0] for r in results if r[2]]
tps_values = [r[1] for r in results if r[2]]
successes = sum(1 for r in results if r[2])
print("\n=== BENCHMARK RESULTS ===")
print(f"Total Successful Requests: {successes}/{CONCURRENT_REQUESTS}")
print(f"Total Wall Time: {total_benchmark_time:.2f} seconds")
if latencies:
print(f"Average Request Latency: {statistics.mean(latencies):.3f}s")
print(f"p50 Latency: {statistics.median(latencies):.3f}s")
print(f"p95 Latency: {statistics.quantiles(latencies, n=20)[18]:.3f}s")
print(f"Average Speed per Stream: {statistics.mean(tps_values):.1f} tokens/sec")
print(f"Combined Processing Capacity: {sum(tps_values):.1f} total tokens/sec")
if __name__ == "__main__":
asyncio.run(main())
Performance Results & Cost Analysis
We tested our setup with Llama-3-8B-Instruct-AWQ (a 4-bit quantized variant of Meta’s model that retains near-perfect evaluation perplexity).
We ran our benchmark tool with a concurrency load of 20 parallel client threads requesting 150 tokens each. The server processed this workload smoothly.
Total Successful Requests: 20/20
Total Wall Time: 4.88 seconds
Average Request Latency: 3.120s
p50 Latency: 3.012s
p95 Latency: 3.410s
Average Speed per Stream: 48.1 tokens/sec
Combined Processing Capacity: 962.0 total tokens/sec
Compare this directly against running a non-quantized model with vanilla Hugging Face pipelines on the same hardware, which consistently choked on CUDA out-of-memory errors once batching rose past 4 concurrent requests.
The Financial Math
Here is the raw cost comparison of our on-prem stack against public API offerings over a 6-month operational window, processing roughly 150 million tokens per month.
| Cost Component | Public API (SaaS) | Budget GPU On-Prem (Our Stack) |
|---|---|---|
| Initial Hardware | $0.00 | $1,650 (2x used 3090, CPU platform) |
| API Costs ($1.50/M tokens) | $1,350 / month | $0.00 |
| Electricity (550W run under load) | $0.00 | ~$45.00 / month |
| Total Cost (Month 1) | $1,350 | $1,695 |
| Total Cost (Month 6) | $8,100 | $1,920 |
By running this hardware locally, we recovered our capital expenditure in the second month of operations.
Lessons Learned & Failures Along the Way
Building your own infrastructure comes with challenges. If you replicate this setup, avoid these three mistakes:
- Power Supply Margins: A single RTX 3090 card spikes to over 350 Watts under load. Two of them, combined with an older Threadripper CPU, pulled 850 Watts. Our first power supply—a standard 850W unit—shut down cleanly during heavy batch runs. Upgrading to a premium 1300W Titanium-rated PSU resolved our stability issues. Do not cut corners on power delivery.
- Thermal Throttling: If your motherboard places your GPUs directly next to each other, the top card cannot draw enough cool air. Under sustained execution, GPU 0 reached 86°C and began throttling its core clocks, causing latency spikes. We fixed this by using an open-air mining frame with high-quality PCIe riser cables, physically separating the cards by six inches.
- Model Selection: Do not run unquantized FP16 models on a budget setup if you want high throughput. The quantized AWQ format is highly optimized for modern CUDA architectures. By utilizing 4-bit weights, we cut our VRAM consumption in half, allowing vLLM to allocate a much larger block of memory to the KV cache. This change increased our maximum concurrent batch size from 8 to 44.