Quantizing a 70B model to fit 2x RTX 3090: AWQ vs GPTQ in practice

Deploying a 70-billion parameter model like Meta-Llama-3-70B on a local workstation is the holy grail for trading desks and quantitative shops looking to parse unstructured alpha signals, financial disclosures, and sentiment feeds without sending sensitive data over third-party APIs. However, the hardware constraints are unforgiving.

A standard FP16 or BF16 representation of a 70B parameter model requires approximately 140 GB of VRAM just to load the weights. This demands an expensive cluster of enterprise-grade H100s or A100s.

To run this locally under a budget, a common setup is a dual consumer-grade GPU workstation containing two RTX 3090 (or 4090) cards, providing a hard ceiling of 48 GB of VRAM. To squeeze a 70B model into this 48 GB footprint while leaving enough headroom for the Key-Value (KV) cache and context windows, we must compress the model down to 4-bit precision.

This post details my hands-on comparison of the two leading 4-bit quantization methodologies: Activation-aware Weight Quantization (AWQ) and Generalized Post-Training Quantization (GPTQ). I cover the technical trade-offs, real-world execution scripts, profiling benchmarks, and the system-level failures I encountered while getting this pipeline operational.

The Core Quantization Pipeline

Quantizing a neural network to 4-bit is not as simple as rounding weights to the nearest lower-precision value. Naive quantization degrades model perplexity to the point of incoherence, especially on complex reasoning tasks. Both GPTQ and AWQ approach this problem differently:

• GPTQ (Generalized Post-Training Quantization): A layer-by-layer optimization framework that minimizes the mean-squared error (MSE) between the output of the original FP16 layer and the quantized layer. It uses second-order Taylor expansion information (Hessian matrices) derived from a calibration dataset to adjust the remaining unquantized weights to compensate for the quantization error of their neighbors.
• AWQ (Activation-aware Weight Quantization): Based on the observation that not all weights are created equal. Only a small fraction (roughly 1%) of weights—the salient weights corresponding to high-magnitude activations—dominate model performance. AWQ identifies these salient weight channels using a calibration dataset, then protects them from quantization error by applying a per-channel scaling factor instead of relying on complex reconstruction.

The high-level data flow of our quantization, compilation, and evaluation loop is structured below:

flowchart TD
 A["FP16 Model Weights"] --> B["Calibration Data Feed"]
 B --> C{"Quantization Selector"}
 C -->|"Observe Activations"| D["AutoAWQ Pipeline"]
 C -->|"Second-Order Taylor"| E["AutoGPTQ Pipeline"]
 D --> F["4-Bit Safetensors"]
 E --> F
 F --> G["vLLM Inference Engine"]
 G --> H["Dual RTX 3090 Execution"]

The Failure Mode: Memory Leaks and Quantization-Time OOMs

My first attempt to quantize Llama-3-70B using AutoGPTQ directly on the dual RTX 3090 workstation was a catastrophic failure. I ran the script with standard parameters, expecting the dual-GPU layout to handle the model split natively via PyTorch’s device_map="auto".

Instead, the process hung for 40 minutes before throwing a fatal Out of Memory (OOM) error:

The underlying issue is that the quantization process itself is highly resource-intensive. During GPTQ calibration, the engine must compute and store the inverse Hessian matrix for every layer. For a 70B parameter model, a single layer’s weight tensor is massive.

When you run quantization with a sequence length of 2048 or 4090 on a calibration set, the intermediate activations and matrix inversions quickly exhaust the remaining VRAM on a 24 GB card, even with basic CPU offloading enabled.

The Fix

To perform 4-bit quantization on a 70B model with a consumer workstation, you cannot run the quantization steps purely in VRAM. You have two options:
1. Rent a high-VRAM cloud instance (e.g., 1x A100 80GB) for an hour to run the quantization step, export the .safetensors model, and copy it to your local dual-3090 workstation for deployment.
2. Force aggressive CPU offloading during quantization, sacrificing days of compute time due to PCIe bottlenecks.

The scripts below assume the cloud/high-VRAM approach for compiling the quantized weights, which is the only viable path for production velocity.

Quantization Implementations

Below are the exact Python pipelines I wrote to compile meta-llama/Meta-Llama-3-70B-Instruct into both 4-bit formats.

1. AutoAWQ Quantization Pipeline

This script loads the unquantized weights, utilizes the pileval dataset (a representative subset of the Pile) as calibration data, and quantizes the model to a 4-bit weight, group-size 128 configuration.

2. AutoGPTQ Quantization Pipeline

For GPTQ, we use the auto_gptq library. We specify a descending order quantization (desc_act=False to allow fast inference on vLLM, as desc_act=True breaks some optimized Triton kernels) and a group-size of 128.

Setting up the High-Speed Inference Server on Dual RTX 3090s

Once you have your 4-bit weights (.safetensors format), you need an inference engine that bypasses standard Hugging Face pipeline bottlenecks. Hugging Face transformers running 4-bit models through raw PyTorch uses naive weight unpacking kernels, which are incredibly slow and run at roughly 2–5 tokens per second.

Instead, we use vLLM, which utilizes custom PagedAttention and optimized AWQ/GPTQ CUDA kernels to maximize throughput.

The GPU Split Configuration

Because each RTX 3090 has only 24 GB of memory, we must split the 70B quantized model across both cards. 4-bit weights for a 70B model occupy roughly 35 GB to 38 GB of space. The remaining ~10 GB is reserved for the KV cache.

Here is the production startup script using vLLM to split the workload evenly across GPU 0 and GPU 1.

For testing the GPTQ version, we modify the execution line to point to our GPTQ weights and change the format parameter:

Performance Profiling: AWQ vs GPTQ

I set up a reproducible benchmarking script to compare the two formats. The script sends concurrent requests to our local vLLM endpoint, testing for Time-to-First-Token (TTFT), generation speed (tokens per second), memory consumption per GPU, and output coherence.

Here is the benchmark runner:

The Hard Numbers

The benchmarks were run locally on a workstation with 2x NVIDIA RTX 3090 Founders Edition GPUs, connected via a standard PCIe Gen 4 x16 bus (no physical NVLink bridge installed).

Key Lessons and Implementation Takeaways

1. AWQ is Faster and Cleaner in Production:
   AWQ systematically outperforms GPTQ in token throughput (14.2 tokens/sec vs 11.8 tokens/sec). The performance delta exists because the GEMM kernels compiled via AWQ avoid scaling overheads during matrix operations. Additionally, AWQ preserves critical features of the base model better, yielding lower perplexity scores than GPTQ on standard evaluation datasets.
2. The NVLink Myth on 3090s:
   Running split-GPU models triggers communication across the PCIe bus (via PyTorch NCCL backend). While installing a physical SLI/NVLink bridge on 3090s theoretically speeds up tensor-parallel communication, in real-world testing, the bottleneck is modern CPU-to-GPU PCIe bandwidth. A PCIe Gen 4 x16 interface is fast enough that the lack of an NVLink bridge introduces less than a 3% latency penalty for a 2-way split.
3. The desc_act=True Trap in GPTQ:
   When compiling GPTQ weights, setting desc_act=True (activation descent) optimizes accuracy metrics slightly. However, this setting breaks the fast Triton execution kernels in vLLM, forcing the engine to fallback to slower, non-optimized kernels. Avoid it if inference speed is your priority.
4. VRAM Budgeting for KV Cache:
   Just because the 4-bit weights fit in 36 GB doesn’t mean you can utilize a 32k context window. At a sequence length of 8,192, the KV cache for a 70B parameter model is huge. vLLM dynamically reserves the remaining VRAM on your GPUs to allocate space for these dynamic variables. If you experience mid-run crashes during long conversations, reduce your --max-model-len to 4096 or lower the --gpu-memory-utilization limit to prevent allocation collisions.