flowchart LR\n A[Service + Java Agent] -->|OTLP head 10%| B[Collector Agent pro Node]\n B -->|loadbalancing nach trace_id| C[Collector Gateway]\n C -->|tail_sampling| D[Tempo]<\/code><\/pre>\nWhy the developers initially ignored the traces<\/h2>\n
After two sprints, half the landscape was instrumented, traces were flowing into the backend, and the sampling costs were under control. Then something happened in the retro that we’ve seen in similar form across several projects: the developers weren’t using the traces.<\/p>\n
Not because they were unusable. But because the span names were cryptic. Auto-instrumentation names spans generically \u2014 an HTTP server span ends up being something like GET<\/code> or, if you’re lucky, GET \/api\/v1\/{id}<\/code>, a JDBC span becomes SELECT shopdb<\/code>. When a developer opens a trace and sees a list of twenty rows of GET<\/code>, SELECT<\/code>, HTTP POST<\/code> with no business context, they close the window again and go back to the log.<\/p>\nThis is the real lesson from the project, and it’s not a lesson about technology: observability rarely fails on the technology; it usually fails on the naming.<\/strong> You can build the cleanest pipeline \u2014 if no one can tell from the trace what’s happening in business terms, the tool won’t get used.<\/p>\nA naming convention you can actually keep up<\/h2>\n
The solution was unspectacular, and precisely for that reason effective. We defined a short convention and enforced it where it pays off most: at the entry points a developer sees first.<\/p>\n
First: service names, business-oriented and consistent. No svc-2<\/code>, no spring-app<\/code>, but checkout-api<\/code>, payment-worker<\/code>, inventory-sync<\/code>. That only costs the discipline of setting OTEL_SERVICE_NAME<\/code> cleanly.<\/p>\nSecond: server spans with the route template instead of the resolved URL. The Java Agent usually does this automatically for Spring, so a thousand variants with different IDs collapse into one span GET \/orders\/{orderId}<\/code> \u2014 which is at the same time the most important lever against a cardinality explosion, that is, against a proliferation of distinguishable span names that bloat the backend.<\/p>\nManual spans only at the business-critical points<\/h2>\n
The third building block is manual spans with meaningful names at the business-critical points. This is the only place where we touched code, and we deliberately kept it small:<\/p>\n
import io.opentelemetry.instrumentation.annotations.WithSpan;\nimport io.opentelemetry.instrumentation.annotations.SpanAttribute;\n\npublic class OrderService {\n\n @WithSpan("order.validate")\n public ValidationResult validate(@SpanAttribute("order.id") String orderId) {\n \/\/ business logic\n }\n}<\/code><\/pre>\nThe @WithSpan<\/code> annotation from opentelemetry-instrumentation-annotations<\/code> creates a span with the given name, @SpanAttribute<\/code> attaches a business value. This puts order.validate<\/code> in the trace instead of an anonymous method signature. A developer investigating a slow checkout sees at a glance whether validation or payment is eating the time.<\/p>\nSince this convention, usage rose measurably \u2014 in the sense that teams opened Tempo on their own to investigate incidents, instead of jumping into the log first. We deliberately did not collect hard before-and-after adoption figures, because they’re easy to fudge. The qualitative signal was unambiguous enough.<\/p>\n
What changed after the first weeks<\/h2>\n
After the first weeks, we had a pipeline that covered the bulk of the Java services, kept the backend load within a predictable range through two-stage sampling, and produced traces that were readable in business terms. The 10 percent baseline rate plus the targeted retention of errors and slow requests reduced the volume in the backend significantly compared to an unfiltered variant \u2014 concretely by more than an order of magnitude, without losing the diagnostically valuable traces.<\/p>\n
More important than the number: the order was right. First auto-instrumentation for breadth, then sampling for cost control, then naming for usability. Dashboards came last \u2014 and were then simple, because the data sat cleanly named in Tempo.<\/p>\n
Limits of this approach<\/h2>\n
Auto-instrumentation covers common libraries, not your business logic. Everything that happens inside a service between two framework calls stays a black box until you set spans manually. Anyone who believes the agent alone gives them full visibility will be disappointed.<\/p>\n
Head-based sampling with a fixed ratio is coarse. On a rarely used service with very low traffic, 10 percent may leave you without a single trace for days. For such services a higher rate, or a per-service rate, is worth it \u2014 which complicates the configuration.<\/p>\n
Tail-based sampling is the most expensive part to operate. It needs memory, it needs the trace_id-based distribution, and it introduces latency through decision_wait<\/code>. In smaller landscapes with manageable volume, head-based sampling alone can be entirely sufficient; then you save yourself the gateway layer. You shouldn’t introduce tail-based sampling on principle, but when the volume forces it.<\/p>\nAnd for languages beyond Java, the comfort is lower. Node still auto-instruments well; with other runtimes more manual work is required. A unified rollout doesn’t mean the same effort everywhere.<\/p>\n
The takeaway: the order decides, not the dashboard<\/h2>\n
A tracing rollout doesn’t start with the pretty dashboard, but with two sober decisions: what you auto-instrument and what you keep. And it stands or falls on the naming \u2014 the best pipeline is useless if no one can tell from the trace what’s happening in business terms.<\/p>\n
We accompany rollouts like this from the first sampling decision to the point where the traces arrive in the teams’ daily work. When your landscape has grown and unified tracing is missing, the structured start is usually the harder part than the technology itself.<\/p>\n","protected":false},"excerpt":{"rendered":"
How an OpenTelemetry rollout starts in a grown service landscape: auto-instrumentation, sampling rules, and span names developers actually use.<\/p>\n","protected":false},"author":6,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[27,1],"tags":[],"class_list":["post-28117","post","type-post","status-publish","format-standard","hentry","category-it-monitoring-en","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/lets-scale.it\/en\/wp-json\/wp\/v2\/posts\/28117","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/lets-scale.it\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/lets-scale.it\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/lets-scale.it\/en\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/lets-scale.it\/en\/wp-json\/wp\/v2\/comments?post=28117"}],"version-history":[{"count":4,"href":"https:\/\/lets-scale.it\/en\/wp-json\/wp\/v2\/posts\/28117\/revisions"}],"predecessor-version":[{"id":28146,"href":"https:\/\/lets-scale.it\/en\/wp-json\/wp\/v2\/posts\/28117\/revisions\/28146"}],"wp:attachment":[{"href":"https:\/\/lets-scale.it\/en\/wp-json\/wp\/v2\/media?parent=28117"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/lets-scale.it\/en\/wp-json\/wp\/v2\/categories?post=28117"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/lets-scale.it\/en\/wp-json\/wp\/v2\/tags?post=28117"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}