Orchestrating data to model to execution in one Temporal workflow

I spent the first half of my career trying to glue three different worlds together: data pipelines (Spark/Airflow), ML serving (FastAPI/Triton), and execution engines (C++, Go, or custom event loops).

In a quantitative trading pipeline, this division of labor is lethal.

Here is the typical failure mode: At 09:30:00, your Airflow DAG kicks off to ingest the latest morning features. At 09:34:15, it succeeds and triggers an API call to your Triton inference server. The model outputs a target portfolio allocation. Your API bridge translates this allocation into a set of orders and pushes them to an execution engine.

Then, reality hits.

At 09:34:18, the broker disconnects. Or your execution engine partially fills the first leg of a spread, and the second leg times out. Or the database deadlock happens during reconciliation. Airflow knows nothing about the live order state. Your execution engine has no context on why the model wanted that specific portfolio vector, so it cannot safely dynamically hedge or adjust limit prices. You are left with a split-brain system where your data state, model state, and broker state are completely out of sync.

We solved this by collapsing the entire boundary. We brought our data ingestion, model evaluation, and execution loop into a single, durably executed system using Temporal.

Here is how we did it, the code that runs it, and the scars we earned along the way.

The Problem: The High-Stakes Sync Gap

Our trading pipeline requires a high degree of coordination. We run a mid-frequency systematic strategy. It recalculates targets every 5 minutes based on order book imbalance, rolling volatility, and macro indicators.

Before adopting Temporal, we used an event-driven architecture built on top of Kafka, Python Celery, and Redis. It looked elegant on a whiteboard, but in production, it was a nightmare of edge cases:

1. The Distributed State Machine Problem: When an order is partially filled and then the network drops, where does that state live? If you store it in Redis, and Redis fails before writing to disk, you double-spend. If you query the broker to reconstruct state, you risk rate limits during high-volatility events.
2. The “Saga” Nightmare: If the model decides to buy 10,000 shares of Asset A and sell 5,000 shares of Asset B, and the buy fills but the sell is rejected, you must roll back or hedge. Writing custom retry, timeout, and rollback logic across five microservices resulted in spaghetti code that was impossible to test locally.
3. The Data-Loss Payload Problem: We were passing large Pandas dataframes through Celery task queues. When the broker brokerages blew up, our Celery workers ran out of memory, killing the worker processes and losing the state of the active trade.

We needed an orchestration engine that treated our code as a single, continuous, durable transaction—one that could sleep for hours, execute sub-second loops, retry flaky API calls with exponential backoff, and guarantee that if our server died mid-execution, another node would pick up the execution exactly where it left off.

The Approach: Unified Temporal Workflows

Temporal allows us to write standard Python code that is guaranteed to run to completion, even if the underlying infrastructure crashes. It achieves this by intercepting calls to “Activities” (side-effect-prone operations like DB writes, API calls, or model predictions) and recording their inputs and outputs to an append-only event log.

If a worker running our trading workflow dies mid-execution, a new worker is spun up, replays the event log to reconstruct the exact state of the workflow variables, and resumes execution without re-running successfully completed activities.

Here is how we structure our unified trading pipeline:

flowchart TD
 A["FetchFeatures"] -->|"DataFrame Reference"| B["RunInference"]
 B -->|"Target Allocations"| C["ExecuteTradesLoop"]
 C -->|"Unfilled Orders"| D["HandlePartialFills"]
 D -->|"Signal Cancel"| E["ReconcilePortfolio"]
 C -->|"All Filled"| E

The Architecture Rules

To make this performant and reliable, we had to establish three strict rules:

1. Never pass raw dataframes in Temporal payloads. Temporal serializes inputs and outputs to its database (PostgreSQL/Cassandra). Passing a 100MB dataframe through workflow parameters will kill your Temporal cluster’s performance. Instead, our feature-generation activity writes data to our internal S3/Parquet store and returns a lightweight URI reference.
2. Separate deterministic workflow logic from non-deterministic activities. All API interactions, ML model evaluations, and system time checks must happen inside Activities. The Workflow itself must be 100% deterministic, relying solely on Temporal’s safe versions of time and random number generation.
3. Use Heartbeats for the execution loop. Our order-matching and execution loop can run for minutes. If the execution worker loses power during this loop, we cannot wait for a standard workflow timeout. The worker must continuously send progress heartbeats back to Temporal.

The Code: The Complete Trading Pipeline

The following is a production-grade implementation of our unified trading pipeline. It uses the temporalio Python SDK to coordinate feature fetching, model evaluation, and execution tracking with automated rollback (the Saga pattern).

1. Defining the Activities (activities.py)

These are our side-effect-prone operations. They interact with our data lake, our PyTorch model, and our broker interface.

2. The Orchestration Workflow (workflow.py)

This is our deterministic workflow. It coordinates the execution sequence, handles data flow safely, monitors order status, and triggers compensating actions if a partial fill or runtime exception occurs.

3. Setting Up the Worker (run_worker.py)

This script bootstraps our execution worker. It registers our workflows and activities, connects to our local or cloud Temporal cluster, and listens to the trading-tasks queue.

4. Triggering the Workflow (trigger_pipeline.py)

This script acts as our Cron scheduler, API gateway trigger, or ad-hoc executor.

Results: Production Under Load

We deployed this Temporal trading pipeline to replace our Celery/RabbitMQ mesh. The system manages strategies executing roughly 3,000 orders per day across three asset classes.

Here is what happened to our operational metrics over a 90-day evaluation period:

1. Tail Latency Recovery

In our old system, when an API timeout occurred, Celery would retry the whole block. This caused our p99 tail latency for order processing to blow out to 12.4 seconds.
With Temporal, we configure individual activity retries with fine-grained control. The local state is retained in the workflow context. Our p99 latency dropped to 345 milliseconds, which is well within our tolerance threshold for mid-frequency execution.

2. Elimination of Split-Brain States

Over 90 days, we faced 4 critical failures:
* Two worker VM restarts due to underlying hypervisor maintenance.
* One complete broker gateway disconnect.
* One database connection exhaustion event during a heavy volatility spike.

In every single case, the pipeline state recovered perfectly:
* The restarted VM workflows resumed instantly from the last logged activity on a fallback worker, without duplicating already completed buy orders.
* The broker disconnect triggered the cancel_and_reconcile compensating action within 5 seconds of the heartbeat timeout, neutralizing our directional exposure.

We had zero double-fills or un-hedged legs during these infrastructure crashes.

3. Log Output Example

Below is a raw log output snippet from our running worker terminal during the partial-fill failure scenario:

Lessons Learned

Collapsing our pipeline boundaries was a massive operational win, but we hit some hard constraints along the way:

1. The Sandbox is Strict

Workflow determinism is absolute. I once tried to use Python’s native random module directly inside the UnifiedTradingWorkflow to stagger order submissions.
On the first worker failure, the replacement worker attempted to replay the event history. Because the native random call returned a different floating-point value on replay, Temporal detected a Non-Deterministic Workflow Change error and instantly locked up the execution loop.
If you need random values, use workflow.random(). If you need timestamps, use workflow.now(). No exceptions.

2. Large State Playloads Kill Performance

During our first integration pass, we tried to return raw trade histories (huge JSON objects containing thousands of microsecond execution events) from our broker activities directly to the workflow.
Our Temporal cluster’s database storage requirements exploded, and workflow startup latency spiked.
We had to shift our pattern: activities write raw, heavy execution logs directly to our Postgres database, and pass only transaction IDs, success statuses, and light vectors back to the orchestration workflow.

3. Worker Topology Matters

We initially ran our machine learning activities on the same workers that ran our broker order loops.
A heavy PyTorch inference model would occasionally saturate the CPU, starving the worker’s thread pool and causing the broker heartbeats to drop.
We quickly learned to split our workers: we now run a model-workers queue on GPU-enabled instances and an execution-workers queue on compute-optimized, low-latency nodes.

By unifying our data, model, and execution states into a single Temporal workflow, we stopped writing code to fight our infrastructure and started writing pure execution logic. Our code is cleaner, our operations are robust, and we no longer wake up to partial-fill nightmares.