The instrumentation is in place, the collector is delivering, and yet the trace for last night’s incident is missing. There’s no error in the application log, none in the collector, and Grafana Tempo reports nothing. This is exactly what the most common operational failure after a tracing rollout looks like: Tempo drops spans at its limits, and it does so quietly. This article shows where that happens, what configuration sits behind it, and which query lets you see it before anyone misses a trace.
Starting point: rollout done, traces still full of gaps
The pattern repeats in most of the tracing rollouts we take over. OpenTelemetry instrumentation across a few dozen services, a collector topology with tail sampling, Grafana Tempo in a Kubernetes cluster with S3-compatible object storage. After the rollout, tracing is a success for a few weeks, the traces are readable, and the teams use them.
Then come the first reports from operations, and they always sound similar. A nightly batch job triggers an incident, but its trace can’t be found. A team sees only part of the spans of a long request in Grafana. And the service graph metrics from the metrics generator — the component of Tempo that computes metrics from spans — break off on some days.
The three symptoms share the same cause: limits that Tempo ships with default values. They fit a test cluster, and in production they kick in without notice, because no client learns about them.
How Grafana Tempo drops spans without saying so
Tempo is built for throughput. The distributor accepts spans via OTLP, distributes them to the ingesters, which form blocks from them, and the compactor stores those blocks in object storage. At several points along this chain there are per-tenant limits, and anyone who exceeds them gets no error in the application log but a counter.
Three limits matter in practice. max_bytes_per_trace limits the size of a single trace; the default value is five megabytes. A batch job that appends thousands of spans to the same trace ID over hours exceeds that; from that point on, all further spans of that trace are dropped. max_traces_per_user limits the number of simultaneously open traces in the ingester, with a default of ten thousand; under high concurrency that is reached quickly.
The third limit is the ingestion rate: rate_limit_bytes and burst_size_bytes cap the bytes per second per tenant. The tenant here is whatever arrives in the X-Scope-OrgID header. Without multi-tenancy, all teams share a single set of limits, and one team with a noisy batch job crowds out the spans of the others.
Every dropped span ends up in tempo_discarded_spans_total, a Prometheus metric with the label reason. It takes values including trace_too_large, live_traces_exceeded, and rate_limited; the Tempo documentation on rejected spans describes the cases. Anyone who doesn’t watch this metric learns of the loss only when someone goes looking for a trace.
The fourth limit sits in the metrics generator
The metrics generator computes service graph and span metrics from spans and writes them via remote write to Mimir or Prometheus. Every combination of service, span name, status, and the configured dimensions is its own time series. The cardinality — that is, the number of distinguishable label combinations — therefore grows with every attribute configured as a dimension.
If resolved URLs with IDs or customer numbers end up as span names or dimensions, endless time series are created. Tempo protects itself with max_active_series per tenant: once the limit is reached, the generator produces no more metrics for new combinations, and the service graph gets gaps without any error appearing. The metric tempo_metrics_generator_registry_active_series shows how close a tenant is to the limit.
The choice of dimensions therefore decides how useful the generator is. Service, span name, status, and the route template are enough for service graph and RED metrics. Everything that differs per user or per order belongs in the trace, not in a metric. Anyone adding a dimension estimates the number of its values beforehand and multiplies it by the existing time series, because that is exactly what the generator does.
The configuration that holds up in production
The solution has three parts: set limits deliberately instead of inheriting defaults, define retention and storage, and alert on losses. The following configuration applies to Grafana Tempo 2.8 in monolithic mode with S3-compatible object storage. The numeric values are examples in the order of magnitude of a mid-sized cluster and must come from your own measurement during the pilot. The structure of the overrides and metrics_generator blocks follows the Tempo configuration reference.
# Grafana Tempo 2.8, monolithic, simplified
storage:
trace:
backend: s3
s3:
bucket: tempo-traces
endpoint: s3.eu-central-1.amazonaws.com
region: eu-central-1
wal:
path: /var/tempo/wal
compactor:
compaction:
block_retention: 336h
metrics_generator:
registry:
external_labels:
source: tempo
storage:
path: /var/tempo/generator/wal
remote_write:
- url: http://mimir-distributor.metrics.svc:8080/api/v1/push
send_exemplars: true
processor:
service_graphs:
dimensions: [deployment.environment]
span_metrics:
dimensions: [http.route]
overrides:
defaults:
metrics_generator:
processors: [service-graphs, span-metrics]
max_active_series: 200000
ingestion:
max_traces_per_user: 50000
rate_limit_bytes: 30000000
burst_size_bytes: 40000000
global:
max_bytes_per_trace: 20000000A block_retention of 336 hours keeps traces for two weeks; after that, the compactor deletes the blocks from object storage, and retention — the storage duration — becomes purely a storage question. max_bytes_per_trace is quadrupled here because long batch traces are wanted from a business perspective; the alternative, cutting the batch job into partial traces, is covered in the section on the limits. The dimensions in the metrics generator are kept deliberately short, and http.route is the route template, never the resolved URL.
send_exemplars: true ensures that the span metrics carry exemplars — that is, measurement points that point directly to a trace. This way, in Grafana, clicking on an outlier in the latency panel leads to the concrete trace without anyone having to search for the trace ID.
An alert that reports the loss before anyone notices it
The most important line in operations is an alert rule on the discarded spans. In the Expression Browser of Prometheus 3.5, one query answers the question of whether spans were lost in the last minute and why:
sum by (tenant, reason) (increase(tempo_discarded_spans_total[1m])) > 0The query returns, per tenant and reason, the number of dropped spans per minute. increase over one minute instead of rate over five yields a readable quantity, spans per minute, and reacts within a minute. As an alert rule with no wait time, the notification goes out as soon as the first span is missing, with the reason in the label. trace_too_large points to the batch jobs, rate_limited to the ingestion limit, live_traces_exceeded to too many simultaneously open traces.
For the metrics generator, a second rule is added that relates tempo_metrics_generator_registry_active_series to the configured limit. If the value climbs above eighty percent, it is time to check the dimensions before the service graph gets gaps. Both rules belong in the dashboard next to the Tempo mixin panels for ingester and compactor, so operations can see the state of Tempo in one place.
What changes with this configuration
With this configuration, the three symptoms from the beginning disappear, because their cause is configured instead of inherited. Long batch traces stay complete, high concurrency no longer runs into max_traces_per_user, and the service graph stays closed. More important is what gets added: every future loss triggers an alert with a reason within a minute, instead of surfacing days later as a missing trace.
We don’t give figures on storage needs, because they depend on span size, sampling rate, and retention and don’t transfer between landscapes. What can be measured and should be measured is the zero: after the change, tempo_discarded_spans_total only rises when someone deliberately hits the limits, and then the reason is in the label.
Where this configuration reaches its limits
Higher limits buy time, not a solution. A max_bytes_per_trace of twenty megabytes makes ingesters and queriers more memory-hungry, and a trace with tens of thousands of spans is no longer readable in Grafana anyway. The cleaner answer to long batch jobs is to start a separate trace per processing step and connect the steps via a shared attribute like batch.run_id. That costs instrumentation work in the code.
The alert on tempo_discarded_spans_total reports losses in Tempo, not before it. Spans that the collector rejects at the memory_limiter, or that tail sampling deliberately drops, don’t show up there. For that you need the collector’s own metrics on rejected and dropped spans, and those belong in the same dashboard.
And the metrics generator doesn’t replace a sampling concept; it only sees what the collector lets through. Anyone who keeps error traces fully in tail sampling and the rest proportionally gets error rates in the service graph that are skewed relative to reality, because errors are overrepresented. This is known and manageable. But it must be told to everyone who reads the RED metrics — that is, rate, errors, and duration per service.
The takeaway: limits are configuration, not a surprise
Grafana Tempo doesn’t lose spans through errors, but through limits that someone didn’t set. Anyone who derives max_bytes_per_trace, max_traces_per_user, the ingestion rate, and max_active_series from their own measurement, sets retention deliberately, and alerts on tempo_discarded_spans_total runs Tempo without silent losses.
We set these four limits in every tracing rollout we take over, before the first day in production, and add the alert the same day. We tie our compensation to your goals, so missed targets have a tangible cost on our side.