OpenTelemetry tracing across Spring services: from zero to useful in a week
Three months ago, our high-frequency order-routing architecture started dropping trades during peak volatility. We run a Spring Boot microservices mesh handling around 14,000 requests per second at peak. When a limit order vanished between our order-gateway, risk-engine, and matching-engine-proxy, our logs were useless. Each service logged its own execution times in isolated CloudWatch log groups. We had timestamps, but clock skew across our EKS nodes made manual correlation impossible.
We needed distributed tracing. I gave myself one week to instrument six core Spring Boot services using OpenTelemetry (OTel), route traces to a Grafana Tempo backend, and extract actionable p99 latency metrics. Here is how I went from zero visibility to pin-pointing a 340ms context-propagation leak in production, along with the exact code and configuration that got us there.
Architecture Overview
Our trade execution path travels across four distinct microservices over HTTP and Kafka. The key challenge was preserving span context across both synchronous REST calls and asynchronous message queues without writing bespoke header-parsing boilerplate in every controller.
flowchart LR client["HTTP Client"] -->|"POST /orders"| gateway["order-gateway"] gateway -->|"gRPC check"| risk["risk-engine"] gateway -->|"Kafka topic"| proxy["matching-engine-proxy"] proxy -->|"OTLP gRPC"| collector["otel-collector"] risk -->|"OTLP gRPC"| collector gateway -->|"OTLP gRPC"| collector collector -->|"Batch export"| tempo["Grafana Tempo"]
Day 1-2: The Auto-Instrumentation Trap
My initial impulse was to inject the OpenTelemetry Java Agent (opentelemetry-javaagent.jar) at runtime via the Dockerfile JAVA_TOOL_OPTIONS.
ENV JAVA_TOOL_OPTIONS=“-javaagent:/opentelemetry-javaagent.jar \
-Dotel.traces.exporter=otlp \
-Dotel.exporter.otlp.endpoint=http://otel-collector:4317 \
-Dotel.resource.attributes=service.name=order-gateway”
This failed catastrophically in our benchmark environment. The Java agent uses byte-code manipulation (ByteBuddy) to instrument every standard library call. Under a 10,000 req/sec load, the order-gateway JVM experienced massive GC pauses. Allocation rate spiked from 120 MB/s to 1.4 GB/s due to unthrottled span creation inside internal Spring framework reactive event loops. p99 latency ballooned from 18ms to 420ms.
We abandoned full auto-instrumentation and switched to programmatic, micrometer-based tracing using micrometer-tracing-bridge-otel combined with opentelemetry-sdk. This gave us precise control over sampler rates and span boundaries.
Day 3: Build Configuration and Context Propagation
To get reliable tracing without GC overhead, we moved to native Spring Boot 3 metrics integration. Here is the clean dependency configuration we established in our parent pom.xml.
<project xmlns=“http://maven.apache.org/POM/4.0.0”
xmlns:xsi=“http://www.w3.org/2001/XMLSchema-instance”
xsi:schemaLocation=“http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd”>
<modelVersion>4.0.0</modelVersion>
<groupId>com.trading.platform</groupId>
<artifactId>service-parent</artifactId>
<version>2.4.0</version>
<packaging>pom</packaging>
<properties>
<java.version>17</java.version>
<spring-boot.version>3.2.3</spring-boot.version>
<opentelemetry.version>1.32.0</opentelemetry.version>
</properties>
<dependencies>
<!– Core Tracing Bridge –>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-tracing-bridge-otel</artifactId>
</dependency>
<!– OTLP Exporter to push spans directly over gRPC –>
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-exporter-otlp</artifactId>
<version>${opentelemetry.version}</version>
</dependency>
<!– AspectJ for @Observed annotations –>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-aop</artifactId>
</dependency>
</dependencies>
</project>
Next, we wrote a explicit OpenTelemetry configuration bean inside order-gateway. This configuration uses a ratio-based sampler (10% of overall traffic) and a BatchSpanProcessor to decouple network I/O from the main execution path.
import io.opentelemetry.api.common.Attributes;
import io.opentelemetry.api.trace.propagation.W3CTraceContextPropagator;
import io.opentelemetry.context.propagation.ContextPropagators;
import io.opentelemetry.exporter.otlp.trace.OtlpGrpcSpanExporter;
import io.opentelemetry.sdk.OpenTelemetrySdk;
import io.opentelemetry.sdk.resources.Resource;
import io.opentelemetry.sdk.trace.SdkTracerProvider;
import io.opentelemetry.sdk.trace.export.BatchSpanProcessor;
import io.opentelemetry.sdk.trace.samplers.Sampler;
import io.opentelemetry.semconv.resource.attributes.ResourceAttributes;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import java.time.Duration;
@Configuration
public class OpenTelemetryConfig {
@Bean
public OtlpGrpcSpanExporter otlpGrpcSpanExporter(@Value(“${otel.exporter.otlp.endpoint:http://otel-collector.monitoring:4317}”) String endpoint) {
return OtlpGrpcSpanExporter.builder()
.setEndpoint(endpoint)
.setTimeout(Duration.ofSeconds(2))
.build();
}
@Bean
public OpenTelemetrySdk openTelemetrySdk(OtlpGrpcSpanExporter exporter,
@Value(“${spring.application.name}”) String serviceName) {
Resource resource = Resource.getDefault()
.merge(Resource.create(Attributes.of(
ResourceAttributes.SERVICE_NAME, serviceName,
ResourceAttributes.DEPLOYMENT_ENVIRONMENT, “production”
)));
SdkTracerProvider tracerProvider = SdkTracerProvider.builder()
.addSpanProcessor(BatchSpanProcessor.builder(exporter)
.setMaxQueueSize(2048)
.setMaxExportBatchSize(512)
.setScheduleDelay(Duration.ofMillis(100))
.build())
.setResource(resource)
.setSampler(Sampler.parentBased(Sampler.traceIdRatioBased(0.10)))
.build();
return OpenTelemetrySdk.builder()
.setTracerProvider(tracerProvider)
.setPropagators(ContextPropagators.create(W3CTraceContextPropagator.getInstance()))
.buildAndRegisterGlobal();
}
}
Day 4: Propagating Traces Over Kafka
Our biggest failure occurred when testing context propagation over Kafka. order-gateway emitted an event, but matching-engine-proxy rendered the event as an isolated root trace rather than a child span.
The traceparent standard header was not being injected into ProducerRecord headers. We resolved this by explicitly wrapping our KafkaTemplate and @KafkaListener container factories with OpenTelemetry interceptors.
Here is the functional code to handle cross-service Kafka tracing:
import io.opentelemetry.api.GlobalOpenTelemetry;
import io.opentelemetry.context.Context;
import io.opentelemetry.context.propagation.TextMapGetter;
import io.opentelemetry.context.propagation.TextMapSetter;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.header.Header;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.stereotype.Component;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.List;
@Component
public class KafkaTracePropagator {
private static final TextMapSetter<ProducerRecord<String, byte[]>> SETTER =
(carrier, key, value) -> {
if (carrier != null) {
carrier.headers().remove(key);
carrier.headers().add(key, value.getBytes(StandardCharsets.UTF_8));
}
};
private static final TextMapGetter<ConsumerRecord<String, byte[]>> GETTER =
new TextMapGetter<>() {
@Override
public Iterable<String> keys(ConsumerRecord<String, byte[]> carrier) {
List<String> keys = new ArrayList<>();
for (Header header : carrier.headers()) {
keys.add(header.key());
}
return keys;
}
@Override
public String get(ConsumerRecord<String, byte[]> carrier, String key) {
Header header = carrier.headers().lastHeader(key);
return header != null ? new String(header.value(), StandardCharsets.UTF_8) : null;
}
};
public void injectContext(ProducerRecord<String, byte[]> record) {
GlobalOpenTelemetry.getPropagators().getTextMapPropagator()
.inject(Context.current(), record, SETTER);
}
public Context extractContext(ConsumerRecord<String, byte[]> record) {
return GlobalOpenTelemetry.getPropagators().getTextMapPropagator()
.extract(Context.current(), record, GETTER);
}
}
We hooked this directly into our service layer to manually enforce execution boundaries across critical trading logic:
import com.trading.platform.gateway.tracing.KafkaTracePropagator;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.context.Scope;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.stereotype.Service;
@Service
public class OrderRoutingService {
private final KafkaTemplate<String, byte[]> kafkaTemplate;
private final KafkaTracePropagator tracePropagator;
private final Tracer tracer;
public OrderRoutingService(KafkaTemplate<String, byte[]> kafkaTemplate,
KafkaTracePropagator tracePropagator,
Tracer tracer) {
this.kafkaTemplate = kafkaTemplate;
this.tracePropagator = tracePropagator;
this.tracer = tracer;
}
public void routeOrder(String orderId, byte[] orderPayload) {
Span span = tracer.spanBuilder(“route_order_kafka_publish”)
.setAttribute(“order.id”, orderId)
.startSpan();
try (Scope scope = span.makeCurrent()) {
ProducerRecord<String, byte[]> record =
new ProducerRecord<>(“orders-v1”, orderId, orderPayload);
// Inject traceparent header into Kafka Record
tracePropagator.injectContext(record);
kafkaTemplate.send(record).whenComplete((result, ex) -> {
if (ex != null) {
span.recordException(ex);
span.setAttribute(“error”, true);
}
span.end();
});
} catch (Exception e) {
span.recordException(e);
span.end();
throw e;
}
}
}
Day 5: Collector Pipeline and Collector Deployment
Do not send spans from microservices directly to your tracing database. If Tempo goes down, client side retries will destroy your Spring application thread pools.
We deployed an OpenTelemetry Collector agent running as a DaemonSet inside our K8s cluster. The collector accepts incoming spans, batches them, drops health check spans, and forwards the filtered data to Grafana Tempo.
Here is our production otel-collector-config.yaml:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
memory_limiter:
check_interval: 1s
limit_percentage: 75
spike_limit_percentage: 20
batch:
send_batch_size: 8192
timeout: 1s
send_batch_max_size: 10240
filter/healthchecks:
error_mode: ignore
traces:
span:
– ‘attributes[“http.target”] == “/actuator/health”‘
– ‘attributes[“http.target”] == “/actuator/prometheus”‘
exporters:
otlp/tempo:
endpoint: “tempo-ingester.monitoring.svc.cluster.local:4317”
tls:
insecure: true
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, filter/healthchecks, batch]
exporters: [otlp/tempo]
To run this pipeline locally for development testing, execute:
-v $(pwd)/otel-collector-config.yaml:/etc/otelcol/config.yaml \
-p 4317:4317 \
-p 4318:4318 \
otel/opentelemetry-collector-contrib:0.95.0
Results and Concrete Profiling Data
By Day 7, we achieved full trace visibility. The total effort transformed our system diagnostics immediately.
Within 24 hours of rolling out the deployment to our staging environment, we captured a trace showing a distinct 340ms dead-time gap inside matching-engine-proxy.
order-gateway: POST /orders |====================| 358ms
├── risk-engine: gRPC /CheckCredit |==| 12ms
└── matching-engine-proxy: Kafka Consume |==================| 341ms
├── [SPAN LEAK] DB Connection Pool Wait |================| 338ms
└── Engine Execute | =| 2ms
——————————————————————————–
The trace made the issue blindingly obvious: matching-engine-proxy had its HikariCP connection pool set to a default size of 10 threads. Under load, incoming Kafka messages were queuing up waiting for a database connection to record audit logs before handing the trade execution off to the core engine.
System Performance Comparison
| Metric | Pre-Instrumentation | Post-Instrumentation (Optimized) |
|---|---|---|
| p99 Latency | 480 ms | 82 ms |
| JVM Heap Allocation Rate | 180 MB/s (baseline) | 194 MB/s |
| Max GC Pause Time (G1GC) | 120 ms | 14 ms |
| Unexplained Order Drops | ~14 per day | 0 |
| Time-to-Diagnosis (MTTR) | 4.5 hours | 3 minutes |
Lessons Learned
- Avoid agent-based auto-instrumentation for low-latency Java. Bytecode manipulation agents inject excessive heap allocations into Spring’s reactive and MVC pipelines. Manual instrumentation using Micrometer Tracing and the
opentelemetry-sdkprogrammatic API yields strict latency budgets. - Always filter health checks at the collector. Actuator polls (
/actuator/health) accounted for 68% of our initial raw span volume. Filtering these out usingfilter/healthchecksin the OpenTelemetry Collector reduced storage costs in Tempo drastically without touching Java code. - Set strict memory limiters on the collector. Without
memory_limiterconfigured, an application trace storm caused by an outage loop can cause your OTel collector to OOMKilled, resulting in complete telemetry loss right when you need it most.