Fine-tuning with LoRA on a single GPU: the data prep that mattered

Last quarter, I tried to adapt a 7B parameter base LLM to process unstructured options trade confirmations and extract structured execution payloads for our downstream OMS. I had a single RTX 4090 (24GB VRAM) running in a spare workstation.

My first three fine-tuning runs using Low-Rank Adaptation (LoRA) were total failures. The loss curves looked fine on paper, but the models hallucinated execution prices, dropped multi-leg option strategies into single-string blobs, and periodically dumped raw system prompts directly into the output stream.

I assumed my rank $r$ was too low, or my learning rate was off. It wasn’t hyperparameter selection; it was data prep.

When you fine-tune with LoRA on a single GPU, parameter efficiency is secondary to data hygiene. LoRA updates a microscopic fraction of the base model’s weight matrices ($<0.1\%$). Because the original model weights remain frozen, any noise, tokenization mismatch, or structural variance in your training data isn’t smoothed out over billions of updated parameters—it gets amplified directly through those tiny low-rank adapter matrices.

Here is the exact data engineering pipeline I built to go from broken extractions to a 99.4% field-level accuracy model running locally on a single consumer GPU.

The setup and the wall

Our input stream consists of mixed-format text payloads from broker chat logs, email fills, and fix-to-text dumps. The goal is to output a single, strictly typed JSON schema ready for execution routing.

On a 24GB GPU, standard full-parameter fine-tuning is out of the question. Even using bitsandbytes 4-bit quantization (QLoRA), memory is tight once you account for sequence length overhead during training backpropagation.

My initial run failed because of three subtle data prep flaws:
1. Dynamic Target Padding: I passed variable-length targets directly into the loss function without masking the prompt tokens. The adapter spent half its capacity learning to predict the prompt text rather than the JSON response.
2. Context Window Exceedance: Inputs longer than 512 tokens were being silently truncated by the Hugging Face tokenizer(..., truncation=True) default settings, dropping the target JSON entirely on complex multi-leg trades.
3. Inconsistent Serialization: Dict keys in the target JSON were non-deterministic. Sometimes action came before qty, sometimes after. The low-rank updates blew up trying to model arbitrary spatial layout variations in the output string.

Here is how the corrected data flow looks from raw unstructured text to low-rank loss computation:

flowchart LR
 rawInput["Raw Chat Text"] --> cleanFn["Clean and Normalize"]
 cleanFn --> jsonCanon["Canonicalize JSON"]
 jsonCanon --> tokSeq["Tokenize Prompt and Target"]
 tokSeq --> maskLoss["Mask Prompt Tokens"]
 maskLoss --> loraModel["QLoRA Forward Pass"]

The Data Engineering Pipeline

To fix this, I abandoned the standard datasets.map() auto-tokenization routines and wrote an explicit, deterministic pipeline using torch.utils.data.Dataset.

Step 1: Canonicalize JSON Targets

Before tokenization, target outputs must be serialized deterministically. Field order must be strictly preserved, white space must be fixed, and numeric formatting must be unified.

Step 2: Custom Tokenization and Prompt Masking

If you train on standard causal language modeling loss without prompt masking, your loss calculation includes predicting every single word of the instruction and input prompt. We only care about the loss on the generated response tokens.

The custom PyTorch Dataset below formats the prompt, tokenizes input and target separately, concatenates them, and builds a strictly defined labels array where prompt tokens are replaced with -100 (the standard PyTorch cross-entropy ignore index).

Running the Training Script

Here is the complete training execution script. It loads Qwen/Qwen2.5-7B-Instruct in 4-bit precision via bitsandbytes, configures the LoRA adapters exclusively on the attention and MLP projections, and runs the training loop using Hugging Face’s Trainer.

When you execute this script, the console output confirms the restricted parameter footprint:

Results and Metrics

The difference between naive training (unmasked prompts, standard Hugging Face dataset auto-padding, non-canonical targets) and structured training (masked prompts, deterministic canonical JSON, explicit sequence length limits) was night and day.

I evaluated both approaches across a validation test set of 250 human-annotated complex multi-leg execution notes.

By masking out prompt tokens from loss calculation, the training loss dropped an order of magnitude faster. Because the model didn’t need to learn to reconstruct the instruction prompts, training converged in 3 epochs instead of 8, cutting single-GPU compute time by more than half.

Lessons Learned

1. Mask your prompt tokens explicitly. If you don’t assign -100 to your system and prompt token positions in labels, your low-rank matrices spend capacity reproducing the prompt syntax. For small rank sizes ($r=8$ or $r=16$), this ruins fine-grained task accuracy.
2. Canonicalize target output text. If target fields fluctuate randomly in output order (e.g., {"a": 1, "b": 2} vs {"b": 2, "a": 1}), loss will spike unpredictably. Write a deterministic pre-processor to sort keys and standardise spaces before passing text to the tokenizer.
3. Use Paged Optimizers for consumer GPUs. Switching optim to "paged_adamw_8bit" prevents sudden Out-Of-Memory (OOM) spikes when sequence lengths approach the ceiling during a batch gradient step.
4. Target all projection matrices. Fine-tuning only q_proj and v_proj resulted in poor JSON schema compliance. Adding k_proj, o_proj, and the MLP blocks (gate_proj, up_proj, down_proj) added minimal VRAM overhead but was critical for output structural integrity.