Kubernetes Gateway API vs Ingress: migrating and what broke

Our production algorithmic trading infrastructure processes thousands of order executions and millions of real-time market data ticks daily. For years, our Kubernetes networking setup relied on a heavily customized Nginx Ingress Controller. It was a classic “annotation soup”—dozens of unreadable annotations per ingress object managing everything from CORS, rate-limiting, and gRPC routing to WebSocket timeouts and Canary weightings.

As our AI-driven alpha generation models expanded, we needed to route gRPC-based ML inference traffic and WebSocket execution feeds reliably without breaking the entire routing layer during a single misconfigured Helm release.

We made the decision to migrate to the Kubernetes Gateway API using Envoy Gateway as our implementation. The transition promised clean role separation, better native traffic management, and an end to our annotation nightmare.

It delivered on those promises, but the migration path was littered with subtle breaks, silent failures, and undocumented behaviors. This is the post-mortem of what broke, how we fixed it, and the concrete code configurations required to get it right.

The Architectural Shift

The fundamental issue with the legacy Ingress resource is its monolithic design. A single developer deploying an app could accidentally break the entire cluster’s routing table by applying an invalid annotation in their local Ingress manifest.

The Gateway API solves this by splitting routing into distinct, role-oriented resources:
* GatewayClass (Infrastructure-level, managed by Platform Engineers)
* Gateway (Entrypoint definition, managed by Network Operations)
* HTTPRoute / GRPCRoute (Routing rules, managed by Application Developers)

This decouples the physical entry point configuration from the application-level traffic rules.

flowchart LR
 client["External Client"]
 gtw["Gateway (Envoy)"]
 rtrest["HTTPRoute (REST API)"]
 rtgrpc["GRPCRoute (ML Inference)"]
 svcrest["REST Service"]
 svcgrpc["gRPC Service"]

 client --> gtw
 gtw -->|"Path Matching"| rtrest
 gtw -->|"Header Matching"| rtgrpc
 rtrest --> svcrest
 rtgrpc --> svcgrpc

The Legacy Ingress Setup: Annotation Soup

To understand why we migrated, here is the raw reality of what our execution Gateway Ingress resource looked like. Note the massive block of Nginx annotations needed just to support WebSockets, gRPC, and custom rate limiting on a single endpoint.

This configuration was brittle. If a junior engineer updated the marketdata-rest-svc paths and forgot to copy the nginx.ingress.kubernetes.io/proxy-read-timeout annotation, our long-lived market data WebSockets would drop silently every 60 seconds.

The Migration Plan and Code

We migrated to Envoy Gateway. First, we had to apply the Gateway API Custom Resource Definitions (CRDs) to our cluster.

The output should look like this:

The Infrastructure Layer: GatewayClass and Gateway

The platform team defines the physical ingress controller instantiation once. No application developers can touch this.

The Application Layer: HTTPRoute and GRPCRoute

With the Gateway established, we split our routing configurations into functional domains. Application teams deployed these alongside their microservices.

Here is the decoupled HTTPRoute for our REST and WebSocket market data feeds:

For our gRPC-based execution services, we leveraged the native GRPCRoute resource (part of the standard Gateway API spec), entirely removing the need for protocol-specific annotations:

What Broke and How We Fixed It

The YAML migration looked elegant on paper. However, our initial deployment to staging triggered immediate regression alerts from our algorithmic execution simulators.

Issue 1: Case-Sensitivity Matching Silent Failures

Our algorithmic trading client programs send specific custom headers to authenticate and prioritize execution streams, notably X-Client-Priority: High.

Under Nginx Ingress, header evaluations were forgiving. When we migrated to the Gateway API and defined an HTTPRoute with a header match rule, our execution routing failed entirely, defaulting to the fallback backend.

Here is the HTTPRoute block that broke:

Why it broke:
The Kubernetes Gateway API specification dictates that HTTP header names are matched in a case-sensitive manner for specific implementations, and conforming proxies like Envoy parse and evaluate all HTTP/2 headers as lowercase. Our trading software was sending X-Client-Priority, but Envoy was matching strictly against x-client-priority or vice versa depending on the transport protocol level.

The Fix:
We had to lower-case all header configurations inside our HTTPRoute definitions to ensure compatibility with Envoy’s HTTP/2 optimization pipeline.

Issue 2: The WebSocket Timeout Trap

During the first 10 minutes of testing the new Envoy Gateway routes, client connections to our order-book update sockets dropped exactly every 15 seconds.

Our legacy Ingress used nginx.ingress.kubernetes.io/proxy-read-timeout: "1800" which kept quiet WebSocket connections alive for 30 minutes without active TCP data frames.

With the Gateway API, there is no standardized spec.rules[].timeout field within the core HTTPRoute spec for idle connections across all Gateway implementations. Envoy Gateway defaults to a standard 15-second request timeout (timeout limit for downstream requests).

The Fix:
To resolve this, we had to define a vendor-specific configuration extension. We used an EnvoyProxy configuration override alongside a ClientTrafficPolicy to set custom TCP keepalive and idle timeout settings.

For the application-level route, we had to define explicit timeouts within our HTTPRoute spec (supported in Gateway API v1 specs via timeouts field):

Setting timeouts.request: 0s is critical. It signals to Envoy that the connection is streaming and should not be aggressively culled when waiting for active execution frames.

Migration Results

The quantitative improvements following the migration to the Kubernetes Gateway API was immediately visible in our observability stack.

Our p99 routing latency dropped dramatically from 12.8ms to 2.4ms. This improvement stems from Envoy’s native HTTP/2 handling and more efficient thread allocation model compared to Nginx’s worker reload model when upstream endpoints change dynamically.

Lessons Learned

1. Spec Differences are Sneaky

Do not assume that feature equivalence exists inside standard specs. Ingress relied heavily on annotations to configure things like cross-origin resource sharing (CORS) and timeouts. In Gateway API, these are either managed by engine-specific custom policies (like Envoy’s ClientTrafficPolicy or BackendTrafficPolicy) or they are deeply integrated into the route specs themselves (like timeouts or filters).

2. Validation Webhooks are Not Optional

Ensure that the Gateway API validating admission webhook is running and healthy in your cluster before applying any HTTPRoute manifests. If the webhook is missing or broken, the API server will accept misconfigured resources, but your data plane will silently drop paths or fail to load-balance correctly without returning descriptive logs.

Output:

3. Header Normalization Matters

When writing HTTP routes to replace legacy setups, normalize headers to lowercase inside your specs. It ensures predictable behaviors across HTTP/1.1 and HTTP/2 transport transformations.