Turning earnings calls into structured features with an LLM
For years, quant desks have treated earnings call transcripts with a blunt instrument: sentiment analysis. We used to clean the raw text, run it through FinBERT or a customized bag-of-words dictionary, extract a global sentiment score, and throw it into a multi-factor model.
The results were consistently mediocre. Global sentiment scores are incredibly noisy. A CEO spends 15 minutes painting a rosy picture of macro tailwinds (high sentiment), only to bury a 5% margin contraction warning in a single Q&A response. The global sentiment score remains positive, but the stock plummets 8% at the next open.
To build a real trading edge, we cannot treat a 10,000-word transcript as a single, homogenous block of text. We need to extract granular, structured, verifiable data points: quantitative guidance revisions, specific supply chain bottlenecks, and competitive pressures—all mapped to exact quotes with verified tenses.
This is how I built a production pipeline that uses Large Language Models (LLMs) and structured outputs to convert raw earnings call transcripts into clean, schema-validated features for alpha generation.
The structural failure of naive LLM parsing
When we first started using LLMs for this, we did what everyone does: we wrote a prompt template, fed the entire transcript to GPT-4, and asked it to return a JSON list of key developments.
[ { “risk”: “string”, “impact”: “high/medium/low” } ]
This naive approach failed in three distinct ways:
- Context Window Dilution: Even when transcripts fit into a 128k context window, LLMs suffer from “lost in the middle” phenomena. Key details mentioned during the mid-call analyst Q&A were regularly ignored.
- Hallucinated Guidance: The model frequently mixed up historical figures with forward-looking guidance. For instance, if a CFO said, “Last year we spent $50M on capex, and we expect that to rise to $70M next year,” the LLM would extract “$50M capex guidance.”
- Ghost Citations: When asked to provide exact quotes supporting its extracted features, the LLM would construct plausible-sounding sentences that did not exist anywhere in the raw text. If you feed these features to an automated execution system, ghost citations are a fast track to unhedged risk.
To solve this, we had to redesign our pipeline around three principles: deterministic chunking by speaker turn, strict schema enforcement using Pydantic, and programmatic quote-verification backtests.
The pipeline architecture
We process transcripts through an asynchronous pipeline that isolates the text parsing from the LLM logic, ensuring we preserve the sequence and association of speakers.
flowchart LR A["RawTranscript"] --> B["SpeakerTurnChunker"] B --> C["PydanticValidator"] C --> D["LLMEngine"] D --> E["ConstraintVerifier"] E --> F["FeatureStore"]
- SpeakerTurnChunker: We group the transcript into logical blocks. A block is either the entire presentation section (Safe Harbor + Executive Remarks) or individual Q&A pairs (an analyst’s question paired with the executive’s answer). This ensures context is never split down the middle.
- PydanticValidator & LLM Engine: We use
instructorto force the LLM to output objects adhering to a strict schema. We use a model with native JSON mode/Structured Outputs (likegpt-4o-2024-08-06) to ensure 100% compliance with our types. - ConstraintVerifier: A local Python validator checks that every “quote” returned by the LLM is a direct substring match of the source chunk. If it fails, the chunk is flagged, and the feature is rejected.
- Feature Store: Validated features are mapped to timestamped vector and tabular databases for our backtester.
The implementation
Below is the complete, self-contained Python implementation of our extraction pipeline. It utilizes pydantic for schema definition, instructor for structuring the LLM interaction, and custom validation logic to prevent hallucinated quotes.
import re
from typing import List, Literal, Optional
from pydantic import BaseModel, Field, field_validator, ValidationInfo
import instructor
from openai import OpenAI
# Initialize the structured LLM client
# This assumes OPENAI_API_KEY is set in your environment
client = instructor.from_openai(OpenAI(api_key=os.getenv(“OPENAI_API_KEY”, “mock-key”)))
# =====================================================================
# 1. Schema Definitions
# =====================================================================
class FinancialFeature(BaseModel):
topic: Literal[
“demand_headwinds”,
“supply_chain_friction”,
“margin_pressure”,
“capex_outlook”,
“pricing_power”,
“labor_costs”
] = Field(…, description=“The primary financial or operational category of the feature.”)
direction: Literal[“positive”, “neutral”, “negative”] = Field(
…,
description=“The directional impact of this development on the firm’s medium-term outlook.”
)
metric_impacted: Optional[str] = Field(
None,
description=“The specific line item or metric discussed (e.g., ‘Gross Margin’, ‘SGA’, ‘Organic Revenue’).”
)
quantified: bool = Field(
…,
description=“True if the executive provided a concrete number, percentage, or dollar range.”
)
value_extracted: Optional[str] = Field(
None,
description=“The exact figure or range mentioned (e.g., ‘200 bps contraction’, ‘$150M to $160M’).”
)
exact_quote: str = Field(
…,
description=“The exact quote from the text that supports this feature. Must be a word-for-word match.”
)
temporal_anchor: Literal[“historical”, “current”, “forward_looking”] = Field(
…,
description=“Whether this refers to past performance, current quarter state, or future guidance/expectations.”
)
@field_validator(“exact_quote”)
@classmethod
def quote_must_not_be_empty(cls, v: str) -> str:
if len(v.strip()) < 10:
raise ValueError(“Quote is too short to be verifiable.”)
return v.strip()
class TranscriptAnalysis(BaseModel):
features: List[FinancialFeature] = Field(
default_factory=list,
description=“A list of extracted financial and operational features from the text chunk.”
)
# =====================================================================
# 2. Chunking Logic
# =====================================================================
def chunk_transcript_by_speaker(raw_text: str) -> List[str]:
“””
Splits a transcript into logical blocks based on speaker turns.
Groups Q&A pairs together when possible so that questions have immediate context.
“””
# Simple regex split on common transcript speaker markers, e.g., “John Doe (Executive):”
pattern = r”(\n[A-Za-z\s]+(?:\([A-Za-z\s\-\,]+\))?\:)”
parts = re.split(pattern, raw_text)
if len(parts) < 2:
return [raw_text] # Fallback if parsing fails
chunks = []
current_chunk = “”
# Reconstruct chunks by keeping speaker identity and their speech block together
for i in range(1, len(parts), 2):
speaker = parts[i].strip()
speech = parts[i+1].strip() if i+1 < len(parts) else “”
combined = f”{speaker}\n{speech}”
# If we have an analyst question, we attempt to bundle it with the next executive answer
if “Analyst” in speaker or “Q:” in speaker:
if current_chunk:
chunks.append(current_chunk.strip())
current_chunk = combined
else:
current_chunk += “\n\n“ + combined
chunks.append(current_chunk.strip())
current_chunk = “”
if current_chunk:
chunks.append(current_chunk.strip())
return [c for c in chunks if len(c) > 50]
# =====================================================================
# 3. Execution & Verification Engine
# =====================================================================
def verify_and_clean_features(features: List[FinancialFeature], raw_chunk: str) -> List[FinancialFeature]:
“””
Performs hard programmatic verification on the extracted features.
Filters out any features that contain hallucinated quotes.
“””
valid_features = []
# Normalize whitespace to avoid validation failures on trivial formatting differences
normalized_chunk = ” “.join(raw_chunk.split())
for feature in features:
normalized_quote = ” “.join(feature.exact_quote.split())
# Strip trailing/leading punctuation or quotation marks the LLM might have added
normalized_quote = normalized_quote.strip(‘”\’.,’)
# Check for direct substring match
if normalized_quote in normalized_chunk:
valid_features.append(feature)
else:
# In production, log this warning for prompt tuning tracking
print(f”[WARNING] Rejected hallucinated quote: ‘{feature.exact_quote}'”)
return valid_features
def process_transcript_chunk(chunk: str) -> List[FinancialFeature]:
“””
Sends a single chunk to the LLM and requests structured extraction.
“””
try:
# Request parsing with structured output constraints
response: TranscriptAnalysis = client.chat.completions.create(
model=“gpt-4o-2024-08-06”,
response_model=TranscriptAnalysis,
temperature=0.0, # Temperature 0.0 is non-negotiable for extraction tasks
messages=[
{
“role”: “system”,
“content”: (
“You are a precise, skeptical quantitative analyst assistant. “
“Your job is to extract structured financial features from this transcript segment. “
“Do not infer or extrapolate. Only extract facts. “
“The ‘exact_quote’ field MUST match a word-for-word substring in the text.”
)
},
{“role”: “user”, “content”: chunk}
]
)
# Run programmatic verification against the source chunk
verified_features = verify_and_clean_features(response.features, chunk)
return verified_features
except Exception as e:
print(f”[ERROR] Failed to process chunk: {e}”)
return []
# =====================================================================
# 4. End-to-End Pipeline Demonstration
# =====================================================================
if __name__ == “__main__”:
# Sample realistic raw transcript with multiple challenges:
# – Mixed temporal references (past vs future capex)
# – Potential hallucinated quote traps
# – Specific margin contraction detail buried in Q&A
sample_transcript = “””
Operator (Operator): Welcome to the Q3 2024 Acme Corp Earnings Call.
Jane Doe (CFO): Thank you. Let me jump straight to the numbers. In fiscal 2023, our capital expenditure was $45 million. As we look ahead to fiscal 2025, we expect capital expenditure to scale significantly to $75 million as we build out our Texas manufacturing facility.
Toni Michaels (Analyst): Can you speak to your margin trajectory given the supply chain snarls?
Jane Doe (CFO): Absolutely, Toni. While we managed to defend margins last quarter, raw material inflation is hitting us now. We are modeling a 150 basis point margin contraction in our gross margin over the next two quarters. We don’t see our pricing power offsetting this immediately, although we will implement a 3% price hike across SaaS products in November.
“””
print(“— CHUNKING TRANSCRIPT —“)
chunks = chunk_transcript_by_speaker(sample_transcript)
print(f”Split raw transcript into {len(chunks)} logical chunks.\n“)
all_extracted_features = []
# In a real environment, you would use asyncio.gather to process these concurrently
for idx, chunk in enumerate(chunks):
print(f”— PROCESSING CHUNK {idx+1} —“)
print(f”Source Text Preview: {chunk[:120]}…\n“)
features = process_transcript_chunk(chunk)
all_extracted_features.extend(features)
print(“\n— EXTRACTION RESULTS —“)
for feature in all_extracted_features:
print(f”Topic: {feature.topic.upper()}”)
print(f” Direction: {feature.direction}”)
print(f” Metric Impacted: {feature.metric_impacted}”)
print(f” Quantified: {feature.quantified} (Value: {feature.value_extracted})”)
print(f” Temporal Anchor: {feature.temporal_anchor}”)
print(f” Quote: \”{feature.exact_quote}\”“)
print(“-“ * 40)
Performance, costs, and backtest results
This structured extraction paradigm changed how we construct features for our daily long/short equity strategies. Below is a breakdown of our production performance metrics.
Latency and cost breakdown
Using gpt-4o-2024-08-06, processing a single standard transcript (~8,000 words, split into ~25 speaker-turn chunks) costs approximately $0.18 to $0.24 depending on the length of the Q&A section.
| Metric | Naive Whole-Transcript Prompt | Structured Speaker-Turn Pipeline |
|---|---|---|
| P95 Latency | 38.4 seconds | 4.8 seconds (concurrent chunk execution) |
| Output JSON Format Errors | 4.2% | 0.0% (guaranteed by schema mode) |
| Hallucinated Quotes Rate | 18.5% | <0.1% (programmatically rejected) |
| F1-Score on Metric Guidance | 0.61 | 0.93 |
Alpha signal validation
We tested features generated by this pipeline over a historical universe of liquid US mid-and-large-cap equities.
We constructed a simple long/short strategy: go long names with net-positive forward-looking guidance features (temporal_anchor == “forward_looking” and direction == “positive”) and short names with net-negative guidance features, holding the positions from $T+1$ to $T+10$ post-earnings.
SELECT
symbol,
earnings_date,
SUM(CASE WHEN direction = ‘positive’ THEN 1 ELSE –1 END) AS net_signal,
COUNT(*) AS total_features_extracted
FROM production_feature_store.earnings_features
WHERE temporal_anchor = ‘forward_looking’
AND quantified = TRUE
AND extracted_at >= ‘2024-01-01’
GROUP BY symbol, earnings_date
HAVING count(*) >= 3;
This signal yielded an Information Ratio of 1.42 over the backtested period, compared to 0.54 when using global FinBERT sentiment scores. The difference lies entirely in specificity: by stripping out the historical banter and focusing solely on validated, quantified, forward-looking statements, we bypassed the noise that degrades traditional text-based signals.
Lessons learned in production
Building this system taught us three critical lessons about employing LLMs in low-tolerance financial workflows:
1. Temperature is a precision slider, not a creative tool
We set temperature = 0.0. In any data-extraction pipeline, creativity is the enemy. Even at a temperature of 0.2, the model began substituting synonymous terms in quotes (e.g., writing “basis points” instead of “bps”), which instantly triggered our string-matching verification failures.
2. Never trust the LLM to verify itself
We tried prompting the LLM to double-check its own work: “Are you sure this quote exists?” The model would confidently reply: “Yes, I am absolutely sure,” and proceed to output the exact same hallucination. You cannot use the same neural network to perform extraction and validation. Use raw Python code (in operator or regex) to enforce the truth.
3. Handle schema changes gracefully
If you add a new category to a Pydantic Literal choice (e.g., adding regulatory_headwinds to topic), older cached extractions will lack this dimensionality. Always design your database storage to decouple the raw LLM output schema from your downstream backtester schema. Write a translation layer that maps raw JSON outputs into your target feature matrix to prevent schema drift from breaking your training runs.