You have a landscape of several dozen services that has grown over years. Each team has its own logging, a handful of dashboards, maybe a Zipkin server somewhere that no one maintains anymore. When a request travels through five services and gets stuck somewhere, someone searches five different log aggregators for a correlation ID that doesn’t even exist in three of them. That’s exactly where the project this article is about started: 40 services, no unified tracing, and a team that knew it needed OpenTelemetry — but had no idea where to begin.
The short answer up front: an OpenTelemetry rollout doesn’t start with dashboards. It starts with auto-instrumentation and with the decision of what you’re going to store in the first place.
Starting point: grown, heterogeneous, no common thread
The client runs around 40 services, most of them in Java (Spring Boot, some still on an old 2.x line), plus a few Node services and two Python batch jobs at the edge. Deployment via Kubernetes, roughly 30 nodes. Metrics already flow through Prometheus, logs land in a central place. Traces effectively didn’t exist — except for two services that had been manually instrumented with Jaeger clients years ago, whose SDK version nobody wanted to touch anymore.
The client’s goal was clearly stated and about as realistic as usual: "We want to see where our requests lose time." That’s a good requirement. It just says nothing about where to start when you have 40 services and a limited budget in the trace backend.
Why the OpenTelemetry rollout doesn’t start at the backend
The obvious reflex with a project like this is to pick a backend, stand up a nice dashboard, and then instrument team by team. That’s the path on which projects get expensive before they get useful.
Traces are high-volume. A single request through five services quickly produces twenty or more spans — a span being the individual operation within a trace, such as an HTTP call or a database query. With a few thousand requests per second across the entire landscape, we’re talking about a lot of time series and spans that have to go somewhere. Dump that into the backend without sampling and you either pay for storage you never read, or the backend tips over under the load.
That’s why the first substantive decision in the project wasn’t the choice of tool, but the question of how much we keep and of what. Trace sampling — the rule for which share of traces actually gets stored — is best defined before the first service sends any data. Otherwise the costs in the backend explode, and you end up retrofitting a brake onto a moving train.
Auto-instrumentation first, because it scales without code changes
With 40 services, manual instrumentation as a starting point is not an option. It would mean adding the SDK to every repository, setting spans by hand, and coordinating that over weeks with individual teams. That takes too long to build momentum.
For the Java services we worked with the OpenTelemetry Java Agent. The agent attaches to the process as a JVM agent and instruments common libraries — servlet containers, JDBC, HTTP clients, messaging — without you changing a single line of application code. We tested the setup with the OpenTelemetry Java Agent 2.10, an OpenTelemetry Collector 0.115, and Grafana Tempo 2.6 as the backend.
The agent comes into the container image as an environment-variable-driven artifact, or is supplied via an init container. The minimal configuration for a service looks like this:
# In the container, before the java -jar call:
export JAVA_TOOL_OPTIONS="-javaagent:/otel/opentelemetry-javaagent.jar"
export OTEL_SERVICE_NAME="checkout-api"
export OTEL_RESOURCE_ATTRIBUTES="service.namespace=shop,deployment.environment=prod"
export OTEL_EXPORTER_OTLP_ENDPOINT="http://otel-collector.observability:4317"
export OTEL_EXPORTER_OTLP_PROTOCOL="grpc"
export OTEL_TRACES_SAMPLER="parentbased_traceidratio"
export OTEL_TRACES_SAMPLER_ARG="0.1"Three things matter here. OTEL_SERVICE_NAME is the name the service appears under in the backend — that becomes relevant later. service.namespace and deployment.environment as resource attributes let you separate prod from staging and group services. And the sampler: parentbased_traceidratio with argument 0.1 means that a service starting a trace keeps it in 10 percent of cases — and that a service continuing an already-sampled trace inherits the parent span’s decision. This is crucial so a trace doesn’t break off midway because a downstream service rolls the dice differently.
Sampling: coarse head-based first, then targeted
Head-based sampling — the decision is made at the start of the trace — has one drawback: at the moment of the decision you don’t yet know whether the request will end in an error. But those are exactly the traces you want to see: the ones with errors and high latency. That’s why the 10 percent rule at the SDK was only set as a first, cheap filter to cap the baseline load.
The actual intelligence went into the collector, via the tail sampling processor. Tail-based sampling — the decision is made after the complete trace has been collected — lets you deliberately keep all erroneous and slow traces and only a sample of the rest:
processors:
tail_sampling:
decision_wait: 10s
num_traces: 50000
policies:
- name: errors
type: status_code
status_code:
status_codes: [ERROR]
- name: slow-requests
type: latency
latency:
threshold_ms: 800
- name: baseline-sample
type: probabilistic
probabilistic:
sampling_percentage: 10
exporters:
otlp/tempo:
endpoint: tempo-distributor.observability:4317
tls:
insecure: true
service:
pipelines:
traces:
receivers: [otlp]
processors: [tail_sampling, batch]
exporters: [otlp/tempo]The catch with tail-based sampling that you need to know: the collector has to buffer all spans of a trace until it can decide. That costs memory and only works cleanly if all spans of a trace arrive at the same collector process. In a horizontally scaled collector fleet you need a layer that routes by trace_id — otherwise each collector only sees part of the trace and decides incorrectly. We solved this with a two-stage collector topology: agent collectors gather per node, and a small gateway layer with the loadbalancing exporter distributes by trace_id to the sampling collectors. This is the part that’s easily underestimated on a first rollout.
flowchart LR
A[Service + Java Agent] -->|OTLP head 10%| B[Collector Agent pro Node]
B -->|loadbalancing nach trace_id| C[Collector Gateway]
C -->|tail_sampling| D[Tempo]Why the developers initially ignored the traces
After two sprints, half the landscape was instrumented, traces were flowing into the backend, and the sampling costs were under control. Then something happened in the retro that we’ve seen in similar form across several projects: the developers weren’t using the traces.
Not because they were unusable. But because the span names were cryptic. Auto-instrumentation names spans generically — an HTTP server span ends up being something like GET or, if you’re lucky, GET /api/v1/{id}, a JDBC span becomes SELECT shopdb. When a developer opens a trace and sees a list of twenty rows of GET, SELECT, HTTP POST with no business context, they close the window again and go back to the log.
This is the real lesson from the project, and it’s not a lesson about technology: observability rarely fails on the technology; it usually fails on the naming. You can build the cleanest pipeline — if no one can tell from the trace what’s happening in business terms, the tool won’t get used.
A naming convention you can actually keep up
The solution was unspectacular, and precisely for that reason effective. We defined a short convention and enforced it where it pays off most: at the entry points a developer sees first.
First: service names, business-oriented and consistent. No svc-2, no spring-app, but checkout-api, payment-worker, inventory-sync. That only costs the discipline of setting OTEL_SERVICE_NAME cleanly.
Second: server spans with the route template instead of the resolved URL. The Java Agent usually does this automatically for Spring, so a thousand variants with different IDs collapse into one span GET /orders/{orderId} — which is at the same time the most important lever against a cardinality explosion, that is, against a proliferation of distinguishable span names that bloat the backend.
Manual spans only at the business-critical points
The third building block is manual spans with meaningful names at the business-critical points. This is the only place where we touched code, and we deliberately kept it small:
import io.opentelemetry.instrumentation.annotations.WithSpan;
import io.opentelemetry.instrumentation.annotations.SpanAttribute;
public class OrderService {
@WithSpan("order.validate")
public ValidationResult validate(@SpanAttribute("order.id") String orderId) {
// business logic
}
}The @WithSpan annotation from opentelemetry-instrumentation-annotations creates a span with the given name, @SpanAttribute attaches a business value. This puts order.validate in the trace instead of an anonymous method signature. A developer investigating a slow checkout sees at a glance whether validation or payment is eating the time.
Since this convention, usage rose measurably — in the sense that teams opened Tempo on their own to investigate incidents, instead of jumping into the log first. We deliberately did not collect hard before-and-after adoption figures, because they’re easy to fudge. The qualitative signal was unambiguous enough.
What changed after the first weeks
After the first weeks, we had a pipeline that covered the bulk of the Java services, kept the backend load within a predictable range through two-stage sampling, and produced traces that were readable in business terms. The 10 percent baseline rate plus the targeted retention of errors and slow requests reduced the volume in the backend significantly compared to an unfiltered variant — concretely by more than an order of magnitude, without losing the diagnostically valuable traces.
More important than the number: the order was right. First auto-instrumentation for breadth, then sampling for cost control, then naming for usability. Dashboards came last — and were then simple, because the data sat cleanly named in Tempo.
Limits of this approach
Auto-instrumentation covers common libraries, not your business logic. Everything that happens inside a service between two framework calls stays a black box until you set spans manually. Anyone who believes the agent alone gives them full visibility will be disappointed.
Head-based sampling with a fixed ratio is coarse. On a rarely used service with very low traffic, 10 percent may leave you without a single trace for days. For such services a higher rate, or a per-service rate, is worth it — which complicates the configuration.
Tail-based sampling is the most expensive part to operate. It needs memory, it needs the trace_id-based distribution, and it introduces latency through decision_wait. In smaller landscapes with manageable volume, head-based sampling alone can be entirely sufficient; then you save yourself the gateway layer. You shouldn’t introduce tail-based sampling on principle, but when the volume forces it.
And for languages beyond Java, the comfort is lower. Node still auto-instruments well; with other runtimes more manual work is required. A unified rollout doesn’t mean the same effort everywhere.
The takeaway: the order decides, not the dashboard
A tracing rollout doesn’t start with the pretty dashboard, but with two sober decisions: what you auto-instrument and what you keep. And it stands or falls on the naming — the best pipeline is useless if no one can tell from the trace what’s happening in business terms.
We accompany rollouts like this from the first sampling decision to the point where the traces arrive in the teams’ daily work. When your landscape has grown and unified tracing is missing, the structured start is usually the harder part than the technology itself.