Serving LangChain agents in production with LangServe

Eight months ago, our algorithmic trading desk transitioned its market sentiment and macro news analysis pipeline from standalone Python scripts to an autonomous LangChain agent. The agent did its job well in backtesting: it queried vector stores for historical fed speeches, scraped real-time financial news RSS feeds, and produced structured sentiment scores that fed into our execution engine.

Then we tried to deploy it.

Our initial microservice wrapper was a handwritten FastAPI layer over the LangChain execution loop. It was a disaster. Streaming token outputs broke whenever a client disconnected prematurely, memory leaks plagued long-running stateful sessions, and schema validation between our pydantic models and LangChain’s internal runnable interfaces required endless boilerplate. When a volatility spike hit the market, our custom endpoints deadlocked under concurrent requests, forcing us to fall back to legacy manual sentiment overrides for three trading sessions.

We replaced that fragile home-brewed bridge with LangServe. Built on top of FastAPI and Pydantic, LangServe automatically exposes LangChain Runnables as REST APIs with native streaming, async execution, and built-in batching support. Here is how we rebuilt our production agent pipeline using LangServe, the architectural traps we fell into along the way, and the exact codebase we run today.

The Problem: The Custom FastAPI Wrapped Agent Trap

When you wrap a complex LangChain agent in a vanilla FastAPI application, you rapidly encounter three major production failure modes:

1. Streaming Asynchrony and Backpressure: LLMs generate tokens incrementally. If your HTTP framework handles SSE (Server-Sent Events) improperly, slow clients will cause your server worker processes to back up, consuming RAM until the worker hits OOM (Out Of Memory) limits.
2. Schema Instability: LangChain tools expect specific inputs. Syncing Pydantic models between client requests, input schemas, output schemas, and intermediate tool execution schemas manual effort yields hundreds of lines of fragile glue code.
3. Execution Context and Cancellation: If a client cancels a request halfway through an agent execution loop, a plain FastAPI endpoint often lets the underlying LangChain agent keep running tool calls, wasting expensive API tokens and database connections.

We attempted to handle this by manually wrapping our agent calls in asyncio.to_thread and hacking custom EventSourceResponse handlers. Under a load test of 50 concurrent requests, our p99 latency spiked to 14.2 seconds, and worker processes regularly ungracefully crashed with starlette.requests.ClientDisconnect exceptions filling Sentry.

The Architecture

LangServe eliminates custom endpoint wiring by wrapping LangChain’s standard Runnable interface directly into FastAPI routing structures. It exposes standard endpoints (/invoke, /stream, /batch, /stream_log) out of the box.

Here is the flow of a client request entering our LangServe-backed trading sentiment microservice:

flowchart LR
 Client["Trading Execution Engine"] -->|"POST /agent/stream_log"| FastAPI["FastAPI App"]
 FastAPI --> LangServe["LangServe Runnable"]
 LangServe --> Agent["LangChain Agent Executor"]
 Agent -->|"Vector Search"| Qdrant["Qdrant Vector DB"]
 Agent -->|"Inference"| LLM["OpenAI Model API"]
 LangServe -->|"SSE Tokens"| Client

The Implementation

Below is the complete, self-contained implementation of our production sentiment agent deployed via LangServe. It implements a multi-tool agent capable of querying market data and evaluating risk factors, wrapped in a production-ready LangServe service.

1. Requirements (requirements.txt)

2. The Microservice Code (server.py)

Launching and Testing the Server

Run the service via uvicorn:

The output confirms that FastAPI and LangServe have successfully mapped the Runnable endpoints:

Requesting Streamed Log Outputs via Python

Client-side integration is straightforward using langserve.RemoteRunnable. This client streams all state shifts, intermediate tool executions, and LLM tokens directly to our trading UI:

Production Results

After deploying this service behind an NGINX reverse proxy on a Kubernetes cluster (2 replicas, 2 vCPU / 4GB RAM limit each), we benchmarked performance under simulated peak volatility loads:

LangServe automatically optimized handling of disconnects via FastAPI’s underlying ASGI request cycle. When a downstream client drops early, the underlying cancellation propagates to the AgentExecutor, instantly halting OpenAI token stream generation and freeing execution slots.

Pitfalls & Hard Lessons Learned

1. Avoid Dynamic Schema Generation in Production: By default, add_routes inspects your LangChain object dynamically to generate Pydantic schemas for OpenAPI documentation. For complex agents with nested tools, this reflection can take 2-5 seconds on startup and consume significant memory. Always explicitly enforce explicit types via .with_types(input_type=..., output_type=...).
2. Beware of Synchronous Tools: If your custom tools execute blocking I/O (e.g., synchronous requests.get instead of httpx.AsyncClient), Uvicorn’s event loop will stall during tool execution. Wrap synchronous tools using asyncio.to_thread inside the tool definition, or use LangChain’s native async tool definitions (async def).
3. Filter stream_log Outbound Traffic: High-frequency streaming endpoints emitting raw stream_log frames can flood client network interfaces with tens of thousands of JSON patch operations (op: add). Aggregate or filter log patches server-side if your clients only need partial tool progress rather than full internal state graph syncs.