Backtesting LLM-generated signals without fooling yourself

We have all seen the cherry-picked backtests on Twitter and LinkedIn. Someone feeds ten years of earnings transcripts or financial news into an LLM, extracts sentiment scores, runs a basic long-short strategy on daily closes, and displays a equity curve with a Sharpe ratio of 3.2.

When I first started integrating LLMs into my systematic trading pipelines, I fell into these exact same traps. My initial backtest of an LLM-driven momentum-sentiment strategy on tech stocks returned an annualized return of 42% with a drawdown of less than 8%. I was ready to hook it up to an interactive Brokers account and retire.

Then I tried to trade it live. Within three weeks, the strategy was down 6.4%, underperforming a simple buy-and-hold benchmark by 800 basis points.

The culprit wasn’t a sudden regime shift. It was a series of subtle, insidious backtesting errors unique to LLM-generated signals: look-ahead bias via API timestamps, ignoring LLM inference latency, and failing to model alpha decay.

Here is how I broke my backtester, how I fixed it, and the production-grade framework I now use to evaluate LLM signals without lying to myself.

The structural traps of LLM backtesting

Using language models to extract trading signals introduces compounding errors that traditional quantitative backtesters (like Backtrader or zipline) are not natively built to catch.

1. The Point-in-Time (PIT) Timestamp Lie

When you query an LLM, or when you use a historical dataset of LLM-scored news, you must be extremely precise about when the information was knowable and when the execution could have actually occurred.

In my failed backtest, I mapped SEC Edgar filing dates directly to daily close prices. If a company filed its 10-Q on October 15th, my backtester assumed I could trade on that information at the October 15th close.

But SEC filings are frequently processed and released after-hours (e.g., 4:05 PM EST) or during volatile trading windows where immediate execution at the “close” price is physically impossible. Furthermore, parsing the document, sending it to the LLM, and getting a structured JSON response takes time.

2. LLM Inference Latency and the Execution Gap

Let’s look at the actual pipeline timeline:

flowchart LR
 A["Raw Text Feed"] -->|"With strict wall-clock ingest time"| B["Signal Engine"]
 B -->|"LLM Inference + API Latency"| C["Point-in-Time Database"]
 C -->|"Strict next-open execution"| D["Backtester Engine"]
 D -->|"Tearsheet & Performance Metrics"| E["Alpha Decay Analysis"]

If an article drops at 09:30:00 AM, and your pipeline takes 12 seconds to download the text, chunk it, send it to a hosted LLM API (like GPT-4o or Claude 3.5 Sonnet), parse the JSON payload, and generate a trade signal, your execution engine cannot trade before 09:30:12 AM.

Twelve seconds doesn’t sound like much, but in liquid equity markets, the institutional algos have already digested the raw text via low-latency NLP models and adjusted the order book. By the time your LLM signal hits the execution queue, the price has moved.

3. Data Leakage via Model Knowledge Cutoffs

If you use an LLM to generate signals on historical data from 2022, but the LLM was trained on data up to 2024, the model knows the future.

Even if you ask it to “ignore all future events,” the underlying weights of the network are already biased by the survival and subsequent performance of those companies. The model is highly likely to classify ambiguous news about Nvidia in 2022 as positive because its weights are thoroughly optimized on Nvidia’s historic run-up in 2023.

My approach: The “Strict Ingest” architecture

To eliminate these biases, I rebuilt my pipeline around three core principles:
1. Physical Timestamping: Every incoming text payload is stamped with a system_ingest_utc timestamp representing when our servers received the packet, and a signal_ready_utc timestamp representing when the LLM returned a valid, parsed JSON signal.
2. Explicit Latency Injection: The backtester shifts the execution timestamp of every trade to $T + \Delta t$, where $\Delta t$ is the empirical latency distribution of the LLM pipeline (typically 2 to 7 seconds for cloud APIs, or 150ms to 400ms for locally hosted 8B parameter models).
3. Decay Sensitivity Analysis: We test the strategy performance across multiple execution delay buckets ($0s, 5s, 30s, 5m, 1h$) to explicitly map out the alpha decay curve of the LLM signals.

The Code: A rigorous point-in-time backtesting engine

The following script implements a vectorised backtester with explicit latency simulation, transaction costs, slippage, and alpha decay calculation. We will simulate a real-world scenario where an LLM is parsing news events to generate trading signals on an intraday basis.

We do not use synthetic, clean mathematical curves; instead, we inject realistic API latency variations and slippage models that scale with volatility.

Results: The brutal curve of LLM signal decay

When we execute the backtester over 15,000 minutes of trading data with our 400 LLM-generated signals, the impact of latency on raw returns is stark. Below is the output generated by running the pipeline code:

Deciphering the numbers

• 0s Latency (The Myth): If we could trade instantly upon the news dropping, we would capture an incredible $14.12\%$ cumulative return across our sample window. This is the false promise of naive backtesting.
• 5s Latency (The Reality of LLMs): In the 5-second delay scenario—representing optimized API calls using structured JSON streaming—our win rate drops from $56.2\%$ to $52.8\%$. Our net cumulative return is slashed by almost half to $8.45\%$.
• 30s Latency (Poor Pipeline Design): If your pipeline suffers from slow parsing, massive prompt overhead, or synchronous API queues, you drop to $1.12\%$ cumulative return. Once transaction fees and slippage are factored in, you are essentially trading noise.
• 300s Latency (The Dead Zone): If you are running cron-jobs that pool and batch documents every 5 minutes, you are actually taking liquidity at the worst possible times. Your performance collapses to $-4.87\%$. The alpha has completely decayed, leaving you holding toxic flow.

Lessons learned: Designing a robust LLM strategy

If you are going to deploy production capital into LLM-driven trading strategies, you must hardcode these structural constraints directly into your evaluation pipeline:

1. Host models locally for critical paths: If you target sub-second signal generation, do not use public cloud endpoints like GPT-4 or Claude. Use locally deployed, quantized open-source models (e.g., Llama-3-8B-Instruct via vLLM) hosted on your own NVMe-equipped GPU servers. This can pull your inference latency down from $3.5$ seconds to under $200$ milliseconds.
2. Never backtest using closing prices: If a signal is generated at 2:15 PM, you must execute against the 2:16 PM or 2:17 PM Open price, incorporating a realistic bid-ask spread and market impact model.
3. Log the wall-clock times: When paper trading or collecting dry-run metrics, log every step of the pipeline. If your data provider delays its websocket stream by even 1.5 seconds, your backtest is invalidated unless that exact delay is modeled historically.