pgvector in production: indexing, recall, and the queries that got slow
We built a semantic search and retrieval system for 12 million financial filings and real-time news articles. Our choice of vector database wasn’t a speci…
Read entryAI Engineering
Jenkins vs GitHub Actions for ML-service CI: what I kept and what I dropped
Two years ago, my team was responsible for maintaining the production deployment pipelines of a suite of real-time machine learning models. These models p…
Read entry
Batching strategy for LLM inference: throughput vs tail latency
During high-volatility market events, such as an unscheduled Federal Reserve rate announcement, my team's algorithmic trading infrastructure processes a m…
Read entry
LangGraph state machines: modeling a multi-step agent as a graph, not a prompt chain
A few months ago, I was tasked with building an automated quantitative research agent. The goal was straightforward: ingest a ticker symbol, pull real-tim…
Read entry
Speculative decoding on vLLM: the latency win and the gotchas
We run a real-time sentiment and market-impact parsing engine that processes news feeds, regulatory filings, and earnings transcripts. Our SLA requires us…
Read entry
Prefix caching in vLLM: cutting TTFB for long system prompts
In high-throughput, LLM-powered trading intelligence pipelines, latency is the bottleneck that kills execution edge. My team runs an automated pipeline th…
Read entry
Building a self-hosted LLM inference stack on a budget
We were spending $4,200 a month on closed-source LLM APIs to power a real-time sentiment extraction and limit-order-book feature pipeline. The setup was s…
Read entry
Monitoring GPU memory fragmentation in long-running inference servers
Six months ago, our team was running a cluster of 64 NVIDIA H100 GPUs hosting a pipeline of custom Stable Diffusion XL and LLaMA-3-70B models. For the fir…
Read entry
LangGraph vs Temporal for AI workflow orchestration
I've shipped agent workflows on both LangGraph and Temporal, and I keep seeing them framed as competitors. They're not. They solve different problems, and…
Read entry
pgBouncer connection pool exhaustion: a real production postmortem
At 2:14 AM the pager went off: every API request was timing out with remaining connection slots are reserved for non-replication superuser connections. Th…
Read entry