Spring Cloud Gateway as the front door: routing, retries, and the timeouts that bit me
When you are routing high-frequency order flows and real-time market data enrichment requests, milliseconds are not just metrics—they are PnL. In our algorithmic trading infrastructure, we positioned Spring Cloud Gateway as the central api gateway at the edge of our kubernetes cluster. It was tasked with token validation, rate limiting, and routing incoming client requests to downstream execution engines.
For months, the gateway ran flawlessly under nominal loads of 5,000 requests per second (RPS). But during a high-volatility market event—the US Consumer Price Index (CPI) print—our edge infrastructure collapsed. The gateway started throwing chain reactions of 504 Gateway Timeout and 500 Internal Server Error, while our downstream execution services sat idle with near-zero CPU utilization.
This is the post-mortem of how default timeout configurations, misconfigured connection pools, and naive retry storms brought down our gateway, and the exact production-grade configurations we implemented to fix it.
The Outage: Anatomy of a Connection Pool Exhaustion
During the CPI release, incoming traffic surged to 24,000 RPS. Within 15 seconds, our p99 latency spiked from a clean 12ms to over 15,000ms. Then, the gateway stopped accepting connections entirely.
When we pulled the thread dumps and TCP socket states from the gateway pods running in Kubernetes, we saw thousands of threads in WAITING states and an explosion of sockets in TIME_WAIT.
The log files were flooded with variations of this stack trace:
at reactor.netty.http.client.HttpClientConnect$HttpIOHandlerObserver.onStateChange(HttpClientConnect.java:438) ~[reactor-netty-http-1.1.10.jar:1.1.10] Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Error Topic: Connection closed prematurely
And immediately following those, we saw connection acquisition failures:
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:264) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
What Actually Happened
Our downstream Execution Service took slightly longer to process trades under heavy load (rising from 8ms to 250ms). Because we had not explicitly configured connection pool limits and timeouts on Spring Cloud Gateway’s underlying Reactor Netty HTTP client, the gateway attempted to spawn new physical connections for every incoming request.
Once the gateway hit the default maximum connection limit (which defaults to elastified limits based on available processors), requests queued up waiting for an available connection from the pool. This triggered a cascading failure:
- The Queue Backlog: Incoming requests waited in the Reactor Netty pool allocation queue.
- Client-Side Timeouts: The external clients timed out at 2,000ms and dropped the connection.
- Orphaned Requests: The gateway, unaware that the client had already disconnected, continued waiting for the downstream connection, then processed the downstream response and tried to write it back to a closed socket.
- The Retry Storm: Our API gateway was configured with a naive retry filter. When a downstream call failed due to a transient network hiccup, the gateway immediately retried the call up to three times. This multiplied our 24,000 RPS load to nearly 70,000 downstream RPS, completely choking our internal Kubernetes network.
The Architecture
To prevent this from happening again, we redesigned our gateway routing topologies. The architecture isolates transient failures at the gateway boundary before they can cascade into our core execution services.
flowchart LR client["External Client"] k8singress["K8s Ingress"] scg["Spring Cloud Gateway"] execService["Execution Service"] mktService["Market Data Service"] client --> k8singress k8singress --> scg scg -->|"Route A: Strict Limits"| execService scg -->|"Route B: Circuit Breaker"| mktService
- External Client: The algorithmic execution clients accessing our API.
- K8s Ingress: NGINX Ingress controller terminates TLS and passes clean HTTP traffic to Spring Cloud Gateway.
- Spring Cloud Gateway: Our edge routing layer, now configured with bounded connection pools, strict timeouts, and backoff-aware retries.
- Execution Service & Market Data Service: Downstream microservices running inside the private Kubernetes overlay network.
The Code: Fixing the Gateways Configuration
Fixing these issues required a two-pronged approach: tuning the underlying Reactor Netty engine via system properties and configuring robust, route-specific timeout and retry behaviors using the Spring Cloud Gateway DSL.
1. Reactor Netty Global Connection Pool Tuning
We first configured the global HTTP client properties in our Spring Boot application.yml. These settings govern the connection pool size, connection acquisition timeouts, and keep-alive behaviors.
cloud:
gateway:
httpclient:
# Time to establish a TCP connection with downstream services
connect-timeout: 500
# Time the gateway will wait for a response from downstream
response-timeout: 1000
pool:
type: FIXED
max-connections: 1000
# Maximum time a request can wait in the queue for an idle connection
acquire-timeout: 250
max-idle-time: 15000
max-life-time: 60000
metrics: true
2. High-Resilience Java-Based Route Configuration
While global timeouts are a good fallback, different downstream services have highly distinct SLA profiles. Our Execution Service requires ultra-low connect timeouts but can tolerate up to 1,500ms processing times during extreme market volatility. Conversely, our Market Data Service should fail fast (e.g., 150ms).
We bypassed YAML-based route declarations in favor of a clean, type-safe Java DSL config. This allowed us to inject fine-grained resilience filters, custom circuit breakers, and retries with exponential backoff and jitter.
Here is the exact routing configuration we deployed to production:
import io.github.resilience4j.circuitbreaker.CircuitBreakerConfig;
import io.github.resilience4j.timelimiter.TimeLimiterConfig;
import org.springframework.cloud.circuitbreaker.resilience4j.ReactiveResilience4JCircuitBreakerFactory;
import org.springframework.cloud.circuitbreaker.resilience4j.Resilience4JConfigBuilder;
import org.springframework.cloud.client.circuitbreaker.Customizer;
import org.springframework.cloud.gateway.filter.ratelimit.KeyResolver;
import org.springframework.cloud.gateway.route.RouteLocator;
import org.springframework.cloud.gateway.route.builder.RouteLocatorBuilder;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.http.HttpStatus;
import reactor.core.publisher.Mono;
import java.io.IOException;
import java.time.Duration;
import java.util.Objects;
@Configuration
public class GatewayRoutingConfig {
@Bean
public RouteLocator customRouteLocator(RouteLocatorBuilder builder) {
return builder.routes()
// 1. High-Priority Trade Execution Route
.route(“execution-route”, r -> r
.path(“/api/v1/trade/**”)
.filters(f -> f
// Strip the prefix before forwarding to downstream
.stripPrefix(1)
// Set route-specific metadata for timeouts (overriding global defaults)
.metadata(“response-timeout”, 1500)
.metadata(“connect-timeout”, 200)
// Retry filter configuration to mitigate transient TCP resets
.retry(retryConfig -> retryConfig
.setRetries(3)
.setSeries(HttpStatus.Series.SERVER_ERROR)
.setStatuses(HttpStatus.TOO_MANY_REQUESTS, HttpStatus.SERVICE_UNAVAILABLE)
.setExceptions(IOException.class, java.net.ConnectException.class)
.setBackoff(
Duration.ofMillis(50), // first backoff
Duration.ofMillis(250), // max backoff
2, // factor
true // enable jitter
)
)
)
.uri(“lb://execution-service”)
)
// 2. Low-Latency Market Data Route with Circuit Breaking
.route(“market-data-route”, r -> r
.path(“/api/v1/market/**”)
.filters(f -> f
.stripPrefix(1)
.metadata(“response-timeout”, 150)
.metadata(“connect-timeout”, 50)
.circuitBreaker(cbConfig -> cbConfig
.setName(“marketDataCircuitBreaker”)
.setFallbackUri(“forward:/fallback/market-data”)
)
)
.uri(“lb://market-data-service”)
)
.build();
}
@Bean
public Customizer<ReactiveResilience4JCircuitBreakerFactory> defaultCustomizer() {
return factory -> factory.configureDefault(id -> new Resilience4JConfigBuilder(id)
.circuitBreakerConfig(CircuitBreakerConfig.custom()
.slidingWindowSize(100)
.failureRateThreshold(50.0f)
.slowCallRateThreshold(50.0f)
.slowCallDurationThreshold(Duration.ofMillis(150))
.waitDurationInOpenState(Duration.ofSeconds(10))
.permittedNumberOfCallsInHalfOpenState(10)
.build())
.timeLimiterConfig(TimeLimiterConfig.custom()
.timeoutDuration(Duration.ofMillis(200))
.build())
.build());
}
@Bean
public KeyResolver userKeyResolver() {
// Rate-limit clients based on the Authorization Header
return exchange -> Mono.justOrEmpty(exchange.getRequest().getHeaders().getFirst(“Authorization”))
.defaultIfEmpty(“anonymous”);
}
}
3. Fallback Controller Implementation
When the Circuit Breaker trips on the Market Data route, we must protect downstream systems by instantly returning cached or degraded payloads rather than letting connections queue up.
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;
import reactor.core.publisher.Mono;
import java.util.Map;
@RestController
public class FallbackController {
private static final Logger log = LoggerFactory.getLogger(FallbackController.class);
@GetMapping(“/fallback/market-data”)
public Mono<ResponseEntity<Map<String, Object>>> marketDataFallback() {
log.warn(“Market Data service circuit breaker triggered. Serving degraded fallback response.”);
Map<String, Object> fallbackPayload = Map.of(
“status”, “DEGRADED”,
“message”, “Real-time feeds are temporarily unavailable. Displaying delayed pricing.”,
“data”, Map.of(“BTCUSD”, 96500.00, “ETHUSD”, 3420.00)
);
return Mono.just(ResponseEntity
.status(HttpStatus.OK)
.header(“X-Fallback-Active”, “true”)
.body(fallbackPayload));
}
}
The Results
After implementing fixed connection pools, aggressive response timeouts, and jittered exponential backoffs, we simulated another CPI-level stress test in our staging Kubernetes cluster.
Using locust, we ran a scenario driving 25,000 RPS against the gateway, while deliberately injecting a 1,000ms network delay on 15% of downstream execution requests.
Here is how the system behaved compared to our previous configuration:
| Metric | Legacy Configuration (Defaults) | Optimized Configuration (With Fixes) |
|---|---|---|
| p95 Latency | 4,200 ms | 68 ms |
| p99 Latency | 15,240 ms | 114 ms |
| HTTP 5xx Error Rate | 24.3% | 0.04% (Transient retries succeeded) |
| Socket State (TIME_WAIT) | 14,000+ | Stable at ~450 |
| Gateway CPU Utilization | 100% (Thread starvation lock) | 34% (Evenly distributed non-blocking IO) |
| Gateway Memory Profile | OOM / Crash Loop | Flat at 410MB |
With the FIXED Reactor Netty connection pool, the maximum active connections to the downstream services never exceeded 1,000. Instead of spawning endless TCP handshakes and clogging up the Linux socket stack, excess connections were queued for up to 250ms and then rejected cleanly via the fallback route, or resolved as soon as a connection returned to the pool.
The exponential backoff with jitter prevented the retry storm. Downstream services were no longer inundated with redundant retries during their brief window of high-latency processing.
Lessons Learned
Operating Spring Cloud Gateway at scale in Kubernetes taught us three fundamental lessons:
- Never rely on default timeouts. The default response timeout for Reactor Netty is infinite. This means a single hanging thread downstream will lock up a gateway connection indefinitely. Always set
connect-timeoutandresponse-timeouton every single route based on historical latency profiles. - Backoff and Jitter are non-negotiable. Retrying instantly on failure will create a self-inflicted Distributed Denial of Service (DDoS) attack. Jitter distributes the load over time, allowing downstream systems breathing room to recover from a brief garbage collection pause or network blip.
- Monitor Reactor Netty Connection Pools. Expose your Netty pool metrics to Prometheus by setting
spring.cloud.gateway.httpclient.pool.metrics: true. Set up alerts on the metricreactor_netty_connection_provider_pending_connections—if this metric rises above zero, your gateway is queuing requests, indicating that your downstream services are slowing down or your pool size is configured too low.