The alert reports elevated response times in checkout, the dashboard confirms it, and then the hunt begins. Filter logs by time window, guess hostnames, copy IDs. Metrics and logs are there \u2014 what’s missing is the path from the incident to the one request that triggered it. Building exactly that path is what OpenTelemetry tracing with Grafana Tempo does, provided four connection points are configured correctly. This article shows those four points, the Grafana configuration behind them, and the click path that ultimately leads from a latency alert to the log line that reveals the cause.<\/p>\n
The picture is the same in most of the environments we take over. Metrics sit in Mimir or Prometheus, logs in Loki, alerting works. During an incident the on-call engineer sees the latency alert, opens three dashboards, and searches Loki with a ten-minute window for anomalies. That search often ends at a plausible explanation rather than at the cause, because the proof is missing that this specific request triggered the alert.<\/p>\n
Tracing is usually there too, but disconnected. Some of the services are instrumented with OpenTelemetry, Tempo or an older Jaeger runs somewhere, and traces can only be found if someone knows a trace ID. No log carries the trace ID, no metric points to a trace, the service graph is empty.<\/p>\n
The goal we set in these projects is concrete. From any alert, the on-call engineer reaches the triggering request and its log lines in three clicks. No ID is searched for, none is copied.<\/p>\n
The chain from incident to request has four connection points, and it can break at any one of them. First, the metric the alert sits on must point to a trace. That is what exemplars do \u2014 measurement points in a metric that carry a trace ID. They originate in the metrics generator, the Tempo component that computes metrics from spans.<\/p>\n
Second, the trace must be complete. Context propagation \u2014 passing the trace ID from call to call \u2014 must work across HTTP, gRPC, messaging, and thread pools. Otherwise the trace ends at the producer, and the cause in the consumer stays invisible. Third, the trace must exist at all. Tail sampling \u2014 selecting the traces to store based on the complete trace \u2014 keeps errors and slow requests in full and discards the rest proportionally.<\/p>\n
Fourth, the logs must carry the trace ID, and their labels must match the attributes of the spans, otherwise the jump from a span into the logs goes nowhere. A common case: the application writes the ID as The data flow is unspectacular. The applications send spans via OTLP to a collector agent per node, which distributes them by trace ID to a collector gateway. There, On the collector side, two settings are decisive. The propagator \u2014 the component that writes and reads the context in headers \u2014 is set to On the Tempo side, the metrics generator must run with the traceId<\/code>, the regex in Grafana looks for trace_id=<\/code>, and no one notices because Loki produces no error for it. None of the four points is a product feature; each is a configuration detail. That is why "we have tracing" so often still means "we’re searching."<\/p>\nWhat OpenTelemetry tracing with Grafana Tempo needs for this<\/h2>\n
tail_sampling<\/code> decides what gets stored, and the export goes to the Tempo distributor. Tempo stores the blocks in object storage, and the metrics generator writes span metrics and service-graph metrics to Mimir via remote write. The examples apply to OpenTelemetry Collector Contrib 0.128, Grafana Tempo 2.8, Loki 3.5, and Grafana 12.1.<\/p>\ntracecontext,baggage<\/code>, so that all services speak the same format. And the sampling rules in the gateway keep status_code: ERROR<\/code> and latencies above a threshold in full. What this pipeline looks like is described in the article on the rollout across 40 services in this series.<\/p>\nservice-graphs<\/code> and span-metrics<\/code> processors and set send_exemplars: true<\/code> on remote write. Without that line there are metrics but no exemplars, and the first click of the path is missing. Mimir must also accept exemplars: the limit max_global_exemplars_per_user<\/code> defaults to zero and must be raised per tenant. The metrics generator is a distinct Tempo component, to which the distributor delivers the spans in parallel to the ingesters; it builds counters and histograms from them and needs its own memory for that, bounded by max_active_series<\/code> in the overrides.<\/p>\nThe Grafana configuration that connects incident and request<\/h2>\n