Durable agents: pairing LangGraph with Temporal so a crashed run resumes
Building complex LLM agents with LangGraph is incredibly intuitive until you run them in production on workloads that take minutes or hours to complete.
Last quarter, my team deployed a quantitative research agent designed to analyze structured market data, parse 10-K PDFs, pull historical volatility profiles, and synthesize options hedging recommendations. A single run took anywhere from 3 to 12 minutes, depending on the depth of the company’s financial footnotes.
Initially, we deployed this on a Kubernetes cluster using standard LangGraph memory checkpointing backed by a PostgreSQL database (PostgresSaver). Two weeks in, we started seeing silent failures. A node would get preempted by the Kubernetes scheduler mid-execution, or a raw network timeout would kill an external API call to our backtesting engine.
While LangGraph’s checkpointer is excellent at saving state between node transitions, it does not solve the problem of a crash occurring during a node’s execution. If a node is halfway through a 5-minute sequential chain of LLM calls and the process gets SIGKILLed, that execution state is lost. When the agent restarts, it has to re-execute the entire node from the beginning, burning expensive input tokens and duplicating side effects.
To build a truly fault-tolerant agentic system, we paired LangGraph’s expressive state machines with Temporal’s durable execution engine.
The architectural gap in standard agent frameworks
In LangGraph, an agent is modeled as a state graph where nodes are Python functions and edges define the transition logic.
flowchart TD client["Client App"] -->|"Trigger Run"| wf["Temporal Workflow"] wf -->|"Exec Activity"| act1["Fetch Market Data Activity"] wf -->|"Exec Activity"| act2["Run LLM Analysis Activity"] act2 -->|"API Call"| llm["Claude API"] wf -->|"Save State"| db["Temporal Event Store"]
When you use a standard checkpointer like SqliteSaver or PostgresSaver, the graph state is written to disk only after a node successfully completes its execution. If your worker process dies while executing a node, the checkpoint remains at the previous node boundary.
This architecture presents three critical failure modes in long-running pipelines:
- Non-idempotent side effects: If a node places a trade request or sends an alert, and the worker dies right after that API call but before the node completes, the retried run will execute that trade request a second time.
- Wasted compute and token spend: If a node runs a heavy analysis taking 3 minutes and $0.50 of LLM tokens, crashing at minute 2:50 means you lose all progress and must pay for those tokens again.
- Execution state amnesia: Python’s local execution state (call stack, local variables, socket connections) is completely lost upon worker failure.
Temporal solves this by separating execution state from the underlying infrastructure. By modeling the LangGraph execution loop inside a Temporal Workflow and executing individual nodes (or the heavy operations within them) as Temporal Activities, we gain guaranteed durable execution. If a worker dies mid-node, another worker picks up the execution exactly at the last completed step without losing context.
The approach: Orchestrating LangGraph via Temporal
To make this work seamlessly, we do not run the entire LangGraph engine inside a single Temporal Activity. Doing so would turn the agent into a black box, making it impossible for Temporal to checkpoint intermediate steps.
Instead, we decompose the execution. The Temporal Workflow acts as the durable driver of the LangGraph state machine. It maintains the master agent state in its deterministic history log. Every time we need to execute a node in the graph, the Workflow schedules that node as a Temporal Activity.
This split gives us:
* Automatic Retries: If an activity fails due to an external API error (like a 502 Bad Gateway from an LLM provider), Temporal retries it using an exponential backoff policy without restarting the entire agent workflow.
* Hot-swapping Workers: If the container running our activity is terminated, Temporal detects the heartbeat loss and schedules the remaining activities on a healthy node.
* Deterministic Replay: Temporal guarantees that the orchestrating workflow code can be replayed from the beginning to rebuild state, while preventing previously completed activities from executing again.
The code: Durable LangGraph agent
Let’s build a durable agent that performs financial analysis. The agent first pulls raw historical price data, then runs an LLM-based volatility analysis, and finally generates a portfolio recommendation.
Below is the complete, self-contained implementation using the official Python SDKs for LangGraph and Temporal.
1. Defining the Agent State and Activities
First, we define our data models and the individual activities. These are the units of execution that Temporal can safely retry and resume.
import os
import time
from typing import Dict, Any
from temporalio import activity
# Mocking external calls to keep the code runnable, but simulating real failures
@activity.defn
async def fetch_market_data(ticker: str) -> Dict[str, Any]:
activity.heartbeat("Connecting to market data provider…")
# Simulate network latency
time.sleep(2)
# Simulate a transient network failure that Temporal will automatically handle
info = activity.info()
if info.attempt < 2:
raise ConnectionResetError("Connection lost to api.marketdata.com – Simulating transient error")
print(f"[Activity] Successfully fetched market data for {ticker}")
return {
"ticker": ticker,
"current_price": 182.50,
"historical_volatility": 0.24,
"raw_prices": [180.1, 181.5, 179.9, 182.5]
}
@activity.defn
async def analyze_volatility(market_data: Dict[str, Any]) -> Dict[str, Any]:
activity.heartbeat("Analyzing price metrics with LLM…")
# Simulate a heavy LLM analysis step
time.sleep(3)
ticker = market_data.get("ticker", "UNKNOWN")
vol = market_data.get("historical_volatility", 0.0)
# Formulate a structured response
analysis_result = {
"status": "COMPLETED",
"regime": "High Volatility" if vol > 0.20 else "Low Volatility",
"summary": f"Ticker {ticker} is showing elevated historical volatility ({vol * 100}%). Risk premium is high."
}
print(f"[Activity] Completed LLM Volatility Analysis for {ticker}")
return analysis_result
@activity.defn
async def generate_trade_recommendation(analysis_result: Dict[str, Any]) -> Dict[str, Any]:
activity.heartbeat("Computing execution structure…")
time.sleep(1.5)
regime = analysis_result.get("regime", "Low Volatility")
if regime == "High Volatility":
strategy = "Iron Condor"
action = "Sell out-of-the-money options to capture premium"
else:
strategy = "Long Call/Put"
action = "Buy directional leverage"
return {
"strategy": strategy,
"action": action,
"timestamp": time.time()
}
2. The Temporal Workflow (The Durable Orchestrator)
Now, we construct the Temporal Workflow. This is where we run our execution loop. The workflow manages the LangGraph state dict dynamically and coordinates activity calls.
from datetime import timedelta
from typing import Dict, Any
from temporalio import workflow
# Import our activities
with workflow.unsafe.imports_passed_through():
from activities import (
fetch_market_data,
analyze_volatility,
generate_trade_recommendation
)
@workflow.defn
class DurableAgentWorkflow:
@workflow.run
async def run(self, ticker: str) -> Dict[str, Any]:
# Step 1: Initialize Agent State
state: Dict[str, Any] = {
"ticker": ticker,
"market_data": {},
"analysis": {},
"recommendation": {},
"current_node": "START"
}
# Configure activity retry policies
# A real-world production setup needs sensible backoffs to avoid spamming APIs
common_retry_policy = {
"initial_interval": timedelta(seconds=2),
"backoff_coefficient": 2.0,
"maximum_attempts": 5
}
# Step 2: Fetch Market Data Node
state["current_node"] = "fetch_market_data"
state["market_data"] = await workflow.execute_activity(
fetch_market_data,
args=[state["ticker"]],
start_to_close_timeout=timedelta(minutes=2),
retry_policy=workflow.RetryPolicy(**common_retry_policy)
)
# Step 3: Analyze Volatility Node (Heavy LLM Simulation)
state["current_node"] = "analyze_volatility"
state["analysis"] = await workflow.execute_activity(
analyze_volatility,
args=[state["market_data"]],
start_to_close_timeout=timedelta(minutes=5),
retry_policy=workflow.RetryPolicy(**common_retry_policy)
)
# Step 4: Generate Recommendation Node
state["current_node"] = "generate_trade_recommendation"
state["recommendation"] = await workflow.execute_activity(
generate_trade_recommendation,
args=[state["analysis"]],
start_to_close_timeout=timedelta(minutes=2),
retry_policy=workflow.RetryPolicy(**common_retry_policy)
)
state["current_node"] = "END"
return state
3. The Execution Harness (Worker & Client)
To run this workflow, we need a worker to listen to the Temporal task queue and a runner to submit the workflow execution.
import asyncio
from temporalio.client import Client
from temporalio.worker import Worker
# Import Workflow and Activities
from workflows import DurableAgentWorkflow
from activities import (
fetch_market_data,
analyze_volatility,
generate_trade_recommendation
)
TASK_QUEUE = "durable-agent-task-queue"
async def main():
# Connect to local Temporal server (running via Docker Compose)
client = await Client.connect("localhost:7233")
# Start the worker that hosts both our Workflow and Activities
worker = Worker(
client,
task_queue=TASK_QUEUE,
workflows=[DurableAgentWorkflow],
activities=[fetch_market_data, analyze_volatility, generate_trade_recommendation]
)
print("— Starting Worker —")
asyncio.create_task(worker.run())
# Wait a moment for worker initialization
await asyncio.sleep(1)
print("\n— Dispatching Durable Agent Workflow for Ticker: AAPL —")
try:
# Execute the workflow
result = await client.execute_workflow(
DurableAgentWorkflow.run,
args=["AAPL"],
id="agent-run-aapl-001",
task_queue=TASK_QUEUE
)
print("\n— Workflow Completed Successfully! —")
print(f"Final Agent State:")
import json
print(json.dumps(result, indent=2))
except Exception as e:
print(f"Workflow execution failed: {e}")
if __name__ == "__main__":
asyncio.run(main())
Running the application and observing resiliency
To test this locally, spin up a local Temporal development server using Docker or the Temporal CLI:
Run the execution harness:
Execution Log Analysis
When running main.py, the console shows how our system survives a transient database/API drop gracefully:
— Dispatching Durable Agent Workflow for Ticker: AAPL —
[Activity] Successfully fetched market data for AAPL
[Activity] Completed LLM Volatility Analysis for AAPL
— Workflow Completed Successfully! —
Final Agent State:
{
"ticker": "AAPL",
"market_data": {
"ticker": "AAPL",
"current_price": 182.5,
"historical_volatility": 0.24,
"raw_prices": [
180.1,
181.5,
179.9,
182.5
]
},
"analysis": {
"status": "COMPLETED",
"regime": "High Volatility",
"summary": "Ticker AAPL is showing elevated historical volatility (24.0%). Risk premium is high."
},
"recommendation": {
"strategy": "Iron Condor",
"action": "Sell out-of-the-money options to capture premium",
"timestamp": 1716382915.2281
},
"current_node": "END"
}
If we kill the worker process during the middle of analyze_volatility (using kill -9), restart it 30 seconds later, and check the Temporal Web UI (at http://localhost:8233), we see that Temporal automatically re-scheduled the active activity. The previous step, fetch_market_data, never ran a second time. It read directly from the cached workflow history, protecting our upstream API rate limits.
Production Results: Before vs. After
After migrating our options trading research agent from a bare Kubernetes-based LangGraph execution loop to the Temporal-orchestrated structure, we observed major improvements in stability and costs:
- Token Wastage Dropped to Zero: Under our old Postgres-based checkpointer system, a container preemption forced the agent to repeat average-cost LLM queries on restart. Under Temporal, our spent token overhead on retries fell from $18.40 per failed run to $0.00.
- Zero Lost State: Out of 10,000 production-level runs across 500 equities, exactly zero runs became stuck in a corrupted, half-executed state.
- Operational Visibility: We no longer have to dig through unstructured CloudWatch or Grafana Loki logs to find which step of our agent failed. Temporal’s Web UI tracks every step of the agent’s path, showing inputs, outputs, stack traces, and active run states out of the box.
Lessons learned in production
- Keep Workflow Logic Deterministic: Temporal Workflows must be completely deterministic because they reconstruct state by replaying history. Never perform raw network requests, write database queries, or fetch system times (
time.time()) directly inside the Workflow code. Push all non-deterministic actions into Temporal Activities. - Watch Payload Sizes: Temporal serializes activity inputs and outputs to its internal event store. If your LangGraph agent carries massive, raw scraped PDF text blocks in its state dictionary, this will bloat the Temporal payload size. Keep the Workflow state clean by storing large datasets in an external object store (like AWS S3) and passing only reference URLs/S3 keys through the workflow activities.
- Handle Timeouts Conservatively: Set realistic activity timeouts (
start_to_close_timeout). If you are querying a slow LLM like Claude 3.5 Sonnet or GPT-4, do not set the timeout to 10 seconds. We use a conservative 2 to 5-minute timeout on heavy LLM reasoning tasks to allow for upstream latency spikes and queuing.