{"id":28119,"date":"2026-04-01T08:00:00","date_gmt":"2026-04-01T06:00:00","guid":{"rendered":"https:\/\/lets-scale.it\/?p=28119"},"modified":"2026-09-16T13:27:51","modified_gmt":"2026-09-16T11:27:51","slug":"observability-project-three-questions-before-you-start","status":"publish","type":"post","link":"https:\/\/lets-scale.it\/en\/observability-project-three-questions-before-you-start\/","title":{"rendered":"Three Questions a Skipped Observability Project Pays for in Months"},"content":{"rendered":"

The agent is installed, the dashboard is open, and for days now someone has been scrolling through metrics hoping something meaningful will jump out. Nothing does. You have visibility into several hundred services but no idea which of them anyone actually cares about\u2014that isn’t monitoring, that’s expensive noise. That’s how our first larger observability engagement went, and the mistake wasn’t a technical one. An observability project loses months when three questions aren’t asked at the start.<\/p>\n

The Starting Point: Hundreds of Services, One Agent, and Not a Single Question<\/h2>\n

The situation is typical of a start with a tool but no mandate. A commercial APM agent was running on several hundred services, complete with call graphs, baselines, and dashboards for everything. Technically it was clean. The agent was easy to operate, the terminology was clear, the data was flowing in.<\/p>\n

What was missing was any notion of which question the system was supposed to answer. Nobody had clarified beforehand which services were critical, which incidents had occurred recently, or what the architecture actually looked like. Without that clarification, every act of instrumentation just produces more surface area to get lost in.<\/p>\n

We’ve seen this pattern many times since, including in projects we ourselves helped run. After two months the monitoring was in place, technically flawless, and yet nobody on the team could say whether the application was currently healthy. Because at the start, nobody had asked what "healthy" even means for this system.<\/p>\n

Why Data Alone Is Not an Answer<\/h2>\n

The assumption behind this, especially among newcomers, is that observability is a tooling question. You buy or build the stack, instrument the application, and then "you have observability." In practice, what you first get is data. And data is not the same as answers.<\/p>\n

The underlying mechanism is simple. Every metric, every trace, and every dashboard is an answer to a question that someone must have asked beforehand. Without the question, you end up with a dashboard nobody looks at and an alert that fires for something nobody cares about. Alerts like these turn into alert fatigue: the desensitization of a team that receives so many irrelevant notifications that it stops noticing the relevant ones too.<\/p>\n

At some point someone mutes the notifications. Then the next real incident hits, and the expensively built monitoring contributes exactly nothing to finding it faster. At that point, management quite rightly asks what the money was spent on.<\/p>\n

Three Questions That Must Be Settled Before the First Instrumentation<\/h2>\n

We no longer walk into any kickoff without these three questions. They sound trivial. Most clients still don’t have the answers ready, and that in itself is a signal.<\/p>\n

The first question is about the current architecture diagram. Not the slide from three years ago, but the real picture: which services talk to which, where synchronous calls run, where something hangs off a database that should have been retired long ago. In nine out of ten cases, no current diagram exists. In that case the first task isn’t instrumentation but drawing this diagram together with the client’s team. That’s uncomfortable, but it’s the map you’re blind without out in the field.<\/p>\n

The second question is about the incidents of recent months. Known incidents show where the system actually breaks, not where it could theoretically break. If the client tells you that once a month a batch job runs off the rails at night and then half the system goes down, it’s clear what needs to become visible first. That way you build monitoring that answers a question someone has already asked\u2014and asked painfully.<\/p>\n

The third question is about the critical points, meaning everything that must never fail. Login, the payment flow, that one interface to a partner system that contracts depend on: that’s where you measure first. Skip this clarification and you instrument the service that sends one email a day with the same depth as the one carrying a thousand requests per second. The overhead then lands in exactly the wrong place\u2014more on that in a separate article in this series.<\/p>\n

The Assessment in Three Queries<\/h2>\n

The answers to the three questions are conversations, but the technical current state can be measured on day one. Where a Prometheus is already running, three queries in the expression browser reveal what the existing monitoring actually covers. The queries are for Prometheus 3.5; the time series up<\/code> and ALERTS<\/code> have existed unchanged for years.<\/p>\n

sum by (job) (up) \/ count by (job) (up)<\/code><\/pre>\n
Prometheus generates the up<\/code><\/a> time series itself for every scrape target: 1 if the target was reachable, 0 if the scrape failed. The query returns, per job, the fraction of reachable targets. A job at 0.6 means 40 percent of its targets haven’t responded since the last scrape, showing which parts of the landscape are already unobserved today.<\/p>\n
count by (alertstate) (ALERTS)<\/code><\/pre>\n
For every active alert, Prometheus keeps a synthetic time series ALERTS<\/code><\/a> with the label alertstate<\/code>, which is either pending<\/code> or firing<\/code>. The query counts both. If there are thirty firing alerts sitting there permanently, alert fatigue isn’t the risk but the state, and the existing rules belong on the examination table before any new instrumentation.<\/p>\n
prometheus_tsdb_head_series<\/code><\/pre>\n
The third query shows the number of active time series in the head block\u2014the order of magnitude of what the existing Prometheus is currently carrying. This number belongs in the kickoff record, because every new instrumentation grows it and because it later shows whether a configuration has spun out of control.<\/p>\n
What Changes When the Questions Are Asked on Day One<\/h2>\n
We’ve experienced both trajectories, and the difference is stark. When the questions are asked only after months, it becomes clear within a week that half the existing instrumentation missed the actual need, and the work starts over. When they’re asked on day one, prioritization almost falls out by itself: known incidents and critical points first, the rest later.<\/p>\n
We deliberately didn’t collect hard numbers on the time saved, because two projects never compare cleanly. Qualitatively, the picture is unambiguous. The three questions cost half a day of conversation at the start; not asking them costs months.<\/p>\n
In practice, the good trajectory looks like this: the first sprint delivers an alerting rule for exactly the incident the client described in the kickoff, and a dashboard for the one critical path\u2014not for everything. Both are tested against the next real incident, not against a checklist. Only once that holds do you add breadth.<\/p>\n
An Observability Project Is a Project, Not a Tool Setup<\/h2>\n
The real lesson from that first engagement wasn’t technical but methodological. Going in unstructured and just looking around doesn’t work, no matter how well you master the tool. What works is treating an observability project like a software project: with an initial assessment, with clear deliverables from the client’s team, and with prioritized work packages. The architecture diagram is supplied by the client, not the consultancy.<\/p>\n
We record the results of the first conversations in writing and translate them into tickets with acceptance criteria. Not out of love for process, but because that breaks the silent assumption that everyone already knows what matters anyway. As soon as a critical point is cast into a ticket with an acceptance criterion, the ambiguities surface\u2014and that’s exactly the right moment for them.<\/p>\n
That’s why our kickoff record always has the same four attachments: the drawn architecture diagram, the list of incidents from the last twelve months with cause and duration, the prioritized list of critical points, and the three measurements from the assessment. If one of them is missing, the project hasn’t started\u2014it has merely begun.<\/p>\n
Where This Approach Hits Its Limits<\/h2>\n
The three questions presuppose that someone can answer them. In organizations where the knowledge about architecture and incidents left with the people who built it, the kickoff delivers only gaps. Then the assessment has to come from the systems themselves\u2014for instance, from traces that make dependencies visible\u2014and that takes longer than half a day.<\/p>\n
Greenfield systems, moreover, have no incident history yet. There, a risk assessment along the critical paths replaces the second question, and it is naturally less reliable than lived experience. And the PromQL-based assessment only works where a Prometheus is already running; with a commercial APM without an open interface, all that remains at this point is the conversation.<\/p>\n
Finally, the three questions are a starting point, not a method for the entire lifetime. They prioritize the first weeks. What comes afterward\u2014for example, the question of how deeply you may instrument before the overhead burdens the system\u2014needs its own rules.<\/p>\n
The Bottom Line: Understanding Comes Before Measuring<\/h2>\n
The success of an observability project isn’t decided between commercial APM and open-source stack, and not in the elegance of the dashboards. It’s decided in the first week, on the question of whether the system was understood before it was measured. Architecture diagram, known incidents, and critical points are the three answers that carry every subsequent decision.<\/p>\n
We learned this the hard way. That’s why clients who bring us in to start an observability project get these three questions and the assessment first, not an agent first.<\/p>\n","protected":false},"excerpt":{"rendered":"
Why an observability project starts with an architecture diagram, known incidents, and critical points\u2014and what the initial assessment looks like in PromQL.<\/p>\n","protected":false},"author":6,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[27,1],"tags":[],"class_list":["post-28119","post","type-post","status-publish","format-standard","hentry","category-it-monitoring-en","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/lets-scale.it\/en\/wp-json\/wp\/v2\/posts\/28119","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/lets-scale.it\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/lets-scale.it\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/lets-scale.it\/en\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/lets-scale.it\/en\/wp-json\/wp\/v2\/comments?post=28119"}],"version-history":[{"count":4,"href":"https:\/\/lets-scale.it\/en\/wp-json\/wp\/v2\/posts\/28119\/revisions"}],"predecessor-version":[{"id":28145,"href":"https:\/\/lets-scale.it\/en\/wp-json\/wp\/v2\/posts\/28119\/revisions\/28145"}],"wp:attachment":[{"href":"https:\/\/lets-scale.it\/en\/wp-json\/wp\/v2\/media?parent=28119"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/lets-scale.it\/en\/wp-json\/wp\/v2\/categories?post=28119"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/lets-scale.it\/en\/wp-json\/wp\/v2\/tags?post=28119"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}