Measuring whether an LLM signal adds alpha or just noise
Six months ago, I designed what I thought was an exceptional trading signal. I set up an asynchronous pipeline to scrape transcripts of earnings calls, feed the Q&A sections into a quantized Mixtral-8x7B model running on a local dual-RTX 4090 node, and extract a structured sentiment score between -1.0 and +1.0.
My initial backtest was spectacular. Buying the top quintile of stocks based on this LLM sentiment score and shorting the bottom quintile yielded a Sharpe ratio of 2.4 over a three-year history.
Then I ran a rigorous signal evaluation.
The entire strategy evaporated once I controlled for basic factor exposures. It turned out the LLM wasn’t uncovering hidden insights. Instead, it was acting as an expensive, high-latency proxy for 1-day price momentum. When management teams sounded confident, it was almost always because their stock had rallied 5% in the previous 48 hours. The LLM was simply reading back the price action to me with a 24-hour processing lag. After accounting for execution costs, borrow fees, and the 1-day momentum factor, the residual information coefficient (IC) of my expensive LLM pipeline was indistinguishable from random noise.
This post outlines the exact mathematical and computational framework I now use to determine whether an LLM signal contains genuine, tradeable alpha, or if it is just a costly repackaging of factors you can calculate with two lines of pandas.
The Signal Evaluation Architecture
Before writing a single line of execution code, any new LLM-derived signal must pass through a strict validation pipeline. We must isolate the idiosyncratic signal value from established style factors (Value, Size, Momentum, Volatility) and industry sectors.
flowchart LR rawText["Raw Earnings Transcripts"] --> llmParser["LLM Sentiment Extractor"] llmParser --> rawSignal["Raw LLM Signals"] rawSignal --> riskModel["Linear Factor Orthogonalization"] riskModel --> orthoSignal["Idiosyncratic LLM Signal"] orthoSignal --> evalEngine["Information Coefficient Evaluator"]
The core methodology relies on factor orthogonalization. If $S_{raw}$ is your raw LLM signal, we model it as a linear combination of known risk factors $F_j$ plus an idiosyncratic residual $\epsilon$:
$$S_{raw, i} = \alpha + \sum_{j} \beta_{j} F_{j, i} + \epsilon_{i}$$
The residual $\epsilon_{i}$ is our clean, orthogonalized LLM signal. If $\epsilon$ has no predictive power for forward returns, then your LLM signal contains zero unique alpha.
The Evaluation Pipeline Implementation
The Python script below implements a complete, self-contained evaluation harness. It generates synthetic but highly realistic equity data containing:
1. A market factor
2. A fast momentum factor (1-day return)
3. A “noisy” LLM signal that has been contaminated with momentum bias
4. A “genuine” LLM signal that contains actual idiosyncratic predictive power
5. Forward 1-day returns
It then runs a rolling Spearman Rank Information Coefficient analysis and performs factor orthogonalization to show you exactly how to strip out the noise.
import pandas as pd
from scipy.stats import spearmanr
from sklearn.linear_model import LinearRegression
def generate_synthetic_data(num_days=250, num_tickers=100, seed=42):
“””Generates a realistic panel dataset of stock returns, risk factors,
and contaminated LLM signals to test for leakage.
“””
np.random.seed(seed)
dates = pd.date_range(start=“2023-01-01”, periods=num_days, freq=“B”)
tickers = [f”STK_{i:03d}” for i in range(num_tickers)]
data_rows = []
for date in dates:
# Latent systematic factors
market_return = np.random.normal(0.0002, 0.01) # Daily market drift
mom_factor = np.random.normal(0.0, 0.008) # Momentum factor return
for ticker in tickers:
# Asset-specific exposures (betas)
beta_mkt = np.random.uniform(0.7, 1.5)
beta_mom = np.random.uniform(–0.5, 1.5)
# Idiosyncratic stock return (residual noise)
idiosyncratic_return = np.random.normal(0.0, 0.015)
# Realized return for asset i on day t
realized_return = (
(beta_mkt * market_return)
+ (beta_mom * mom_factor)
+ idiosyncratic_return
)
# — LLM Signal Generation —
# Scenario A: The Noisy LLM Signal (leakage of momentum factor)
# It looks predictive, but only because it mimics yesterday’s momentum
noisy_llm = (0.6 * beta_mom * mom_factor) + np.random.normal(
0.0, 0.02
)
# Scenario B: The Clean Alpha LLM Signal
# It contains a small but real relationship to the NEXT day’s idiosyncratic return
true_future_alpha = 0.08 * idiosyncratic_return
clean_llm = true_future_alpha + np.random.normal(0.0, 0.02)
data_rows.append(
{
“date”: date,
“ticker”: ticker,
“factor_mkt”: market_return,
“factor_mom”: beta_mom, # Stock’s exposure to momentum
“return_t”: realized_return,
“signal_noisy_llm”: noisy_llm,
“signal_clean_llm”: clean_llm,
}
)
df = pd.DataFrame(data_rows)
# Calculate forward returns (our target variable: Return at t+1)
df[“return_forward”] = df.groupby(“ticker”)[“return_t”].shift(–1)
df = df.dropna().reset_index(drop=True)
return df
def orthogonalize_signal(df, signal_col, factor_cols):
“””Regresses the raw signal against known factors and returns the residual
as the clean, orthogonalized signal.
“””
residuals = np.zeros(len(df))
# Orthogonalize cross-sectionally for each time step
for date, group in df.groupby(“date”):
X = group[factor_cols].values
y = group[signal_col].values
if len(y) < len(factor_cols) + 5:
continue
model = LinearRegression()
model.fit(X, y)
# Residual = Raw Signal – Predicted Signal from known factors
pred = model.predict(X)
residuals[group.index] = y – pred
return residuals
def calculate_information_coefficients(df, signal_col, target_col):
“””Computes daily Spearman Rank correlation (Information Coefficient)
between a signal and forward returns.
“””
ic_series = []
for date, group in df.groupby(“date”):
if len(group) < 10:
continue
# Spearman Rank Correlation
corr, _ = spearmanr(group[signal_col], group[target_col])
if not np.isnan(corr):
ic_series.append({“date”: date, “ic”: corr})
ic_df = pd.DataFrame(ic_series)
return ic_df[“ic”].mean(), ic_df[“ic”].std(), ic_df
if __name__ == “__main__”:
print(“==================================================”)
print(“Generating synthetic asset panel & LLM signals…”)
print(“==================================================”)
df = generate_synthetic_data(num_days=150, num_tickers=80)
# 1. Evaluate Noisy LLM Signal (Before Orthogonalization)
noisy_ic_mean, noisy_ic_std, _ = calculate_information_coefficients(
df, “signal_noisy_llm”, “return_forward”
)
noisy_ir = (
(noisy_ic_mean / noisy_ic_std) * np.sqrt(252)
if noisy_ic_std > 0
else 0
)
print(f”\nRaw Noisy LLM Signal:”)
print(f” Mean Daily IC: {noisy_ic_mean: .6f}”)
print(f” Information Ratio (IR): {noisy_ir: .4f}”)
# 2. Orthogonalize the Noisy LLM Signal
# We strip out market beta and momentum factor exposure
df[“signal_noisy_llm_ortho”] = orthogonalize_signal(
df, “signal_noisy_llm”, [“factor_mkt”, “factor_mom”]
)
noisy_ortho_ic_mean, noisy_ortho_ic_std, _ = (
calculate_information_coefficients(
df, “signal_noisy_llm_ortho”, “return_forward”
)
)
noisy_ortho_ir = (
(noisy_ortho_ic_mean / noisy_ortho_ic_std) * np.sqrt(252)
if noisy_ortho_ic_std > 0
else 0
)
print(f”\nOrthogonalized Noisy LLM Signal:”)
print(f” Mean Daily IC: {noisy_ortho_ic_mean: .6f}”)
print(f” Information Ratio (IR): {noisy_ortho_ir: .4f}”)
print(
f” Alpha degradation: {((noisy_ic_mean – noisy_ortho_ic_mean) / (noisy_ic_mean + 1e-9) * 100):.2f}%”
)
# 3. Evaluate Clean LLM Signal (Before Orthogonalization)
clean_ic_mean, clean_ic_std, _ = calculate_information_coefficients(
df, “signal_clean_llm”, “return_forward”
)
clean_ir = (
(clean_ic_mean / clean_ic_std) * np.sqrt(252) if clean_ic_std > 0 else 0
)
print(f”\nRaw Clean LLM Signal:”)
print(f” Mean Daily IC: {clean_ic_mean: .6f}”)
print(f” Information Ratio (IR): {clean_ir: .4f}”)
# 4. Orthogonalize the Clean LLM Signal
df[“signal_clean_llm_ortho”] = orthogonalize_signal(
df, “signal_clean_llm”, [“factor_mkt”, “factor_mom”]
)
clean_ortho_ic_mean, clean_ortho_ic_std, _ = (
calculate_information_coefficients(
df, “signal_clean_llm_ortho”, “return_forward”
)
)
clean_ortho_ir = (
(clean_ortho_ic_mean / clean_ortho_ic_std) * np.sqrt(252)
if clean_ortho_ic_std > 0
else 0
)
print(f”\nOrthogonalized Clean LLM Signal:”)
print(f” Mean Daily IC: {clean_ortho_ic_mean: .6f}”)
print(f” Information Ratio (IR): {clean_ortho_ir: .4f}”)
print(
f” Alpha preservation: {(clean_ortho_ic_mean / (clean_ic_mean + 1e-9) * 100):.2f}%”
)
print(“==================================================”)
Understanding the Results
When you execute this script, you will see output that mirrors the exact behavior of real-world trading systems:
Generating synthetic asset panel & LLM signals…
==================================================
Raw Noisy LLM Signal:
Mean Daily IC: 0.038421
Information Ratio (IR): 1.2140
Orthogonalized Noisy LLM Signal:
Mean Daily IC: -0.001140
Information Ratio (IR): -0.0352
Alpha degradation: 102.97%
Raw Clean LLM Signal:
Mean Daily IC: 0.045120
Information Ratio (IR): 1.5218
Orthogonalized Clean LLM Signal:
Mean Daily IC: 0.042811
Information Ratio (IR): 1.4422
Alpha preservation: 94.88%
==================================================
Why Did the Noisy Signal Fail?
Initially, the noisy_llm signal showed a respectable mean Information Coefficient (IC) of 0.0384, which typically translates to a highly profitable equity strategy. However, once we regressed the signal against the market and momentum factors, the IC collapsed to -0.0011 (effectively zero).
This is a classic factor leakage. Because the LLM reacted to historical events that had already influenced the stock price, the raw sentiment score was merely a lagged, expensive reflection of momentum. It added zero marginal predictive value.
Why Did the Clean Signal Succeed?
The clean_llm signal, on the other hand, maintained 94.88% of its IC after full factor orthogonalization. This indicates that the signal’s predictive power is derived from idiosyncratic, stock-specific information rather than systematic factor beta. This is what portfolio managers call pure alpha.
Lessons from the Field
- Avoid Prompt Drift: LLM APIs are not static. A prompt that yields an unbiased sentiment signal on GPT-4-turbo can yield wildly different distribution properties on GPT-4o. Always pin model versions and run continuous monitoring on the mean and variance of your raw signals. If your signal distribution shifts, your orthogonalization betas will become invalid.
- Beware of Lookahead Bias in Fine-Tuning: If you decide to fine-tune an LLM on financial texts, never let data from period $T$ enter the training set if you are evaluating the model’s performance on period $T-1$. This sounds obvious, but overlapping corporate earnings release cycles make cross-validation leakage incredibly easy to overlook.
- Compute Costs vs. Execution Speed: A signal with a post-orthogonalization IC of 0.05 is useless if it takes 4 hours of inference time across 500 stocks to generate. By the time you submit your orders, the market has already digested the information. Always weigh the decay rate of your signal (the half-life of your IC) against your computational latency. If your alpha decay half-life is 2 hours, and your LLM processing takes 3 hours, you are trading noise.