Metrics, logs, and traces, in that order

The pager goes off at 3am. The alert says error rate is above threshold. This is the moment every observability investment pays for itself or fails to, and the quality of the next hour is determined entirely by what the team built before this moment. The three words that describe the field, metrics, logs, and traces, are usually presented as a menu, pick your favorite. They are not a menu. They are a sequence, and the sequence has a reason.

Metrics tell you something is wrong. Logs tell you what is wrong. Traces tell you where it is wrong. Each one answers a different question, and you need the answers in that order. The teams that skip the first question are the ones that spend the 3am hour reconstructing the answer from partial evidence, and the teams that built all three can go from pager to root cause in minutes instead of hours.

Metrics: the question "is it broken"

A metric is a number over time: requests per second, error rate, latency percentiles, queue depth, CPU, memory. The power of metrics is that they are cheap to store, easy to chart, and perfect for alerting, because an alert is a rule about a number. The 3am pager exists because a metric crossed a threshold.

The art of metrics is choosing which numbers matter, and the discipline is the four golden signals. Latency tells you how long requests take, and the percentile matters more than the average, because the p95 and p99 are where the unhappy users live. Traffic tells you how much work is coming in, and it is the context for every other number, because an error rate of one percent during a traffic spike is a different story than during a lull. Errors tell you the failures, and they should be measured as a rate, not a count. Saturation tells you how full the system is: how close the database is to its connection limit, how full the queue is, how close the CPU is to busy. Saturation is the leading indicator, the number that moves before the errors start, and the teams that watch it catch the problem before the users do.

The other metric discipline is the RED and USE distinction, which is the same idea from two angles. For every service, you want the request level numbers, the rate, errors, and duration, and the resource level numbers, the utilization, saturation, and errors of the underlying machines. The two views catch different problems, and the teams that only monitor one are blind to the other.

Logs: the question "what happened"

When the metric says error rate is up, the log is the first place to look for the why. Logs are the record of what actually happened: the request, the failure, the stack trace, the message that the code wrote when it could not do what it was asked. Logs are the raw material of the investigation, and their quality determines how fast the investigation goes.

The log quality problem is the one nobody budgets for. A log line that says "error: something failed" is a log line that will be read at 3am and cursed at. The log lines that save the hour are the ones with context: the request ID, the user ID, the service name, the operation, the specific failure. The discipline of structured logging is the discipline of writing the investigation for your future self, because the future self is always you, and it is always 3am.

The second log discipline is knowing what not to log. The log volume problem is real: a service that logs everything logs nothing, because the signal is buried in the noise, and the storage cost grows with every line. The teams that log too much discover that their logs are a haystack, and the teams that log too little discover they have nothing to search. The balance is the same balance as metrics: log the things that change the investigation, and skip the things that will never be read.

Traces: the question "where"

The trace is the answer to the distributed question that neither metrics nor logs can answer alone. A request crosses five services, and the metric says the whole thing is slow, and each service has its own logs, and the question is which service is the bottleneck. The trace is the map: it shows the request's path, the time spent in each service, and the boundaries between them.

The trace is where the modern debugging story lives, because modern systems are distributed by default. A monolith has one set of logs and the investigation is contained. A system of services has logs scattered across processes and machines, and without the trace ID that ties the request together, the investigation is a guessing game across five different log systems.

The investment in traces has a specific shape: a trace ID generated at the entry point, propagated through every call, and attached to every log line. The teams that do this get the 3am win, the request that is slow, and the trace that shows exactly which service and which call took the time. The teams that do not get the log spelunking.

The sequence is the strategy

The reason the sequence matters is that each layer makes the next one possible. The alert is a metric, and it is the entry point, the thing that wakes you up. The investigation starts with logs, the record of what happened, and the trace ID in the log is what ties the logs of different services together. The trace is the map that shows the path. The teams that have all three can go from pager to root cause in the time it takes to read a dashboard and follow a trace. The teams that have one are the teams that can detect the problem and not diagnose it.

The teams that have none are the teams that find out about the outage from a user. The user is a terrible monitoring tool, and the only thing worse than finding out from the user is not finding out at all.

Start with the question, not the tool

The practical advice is to start with the question you need answered, and buy or build the tool that answers it. The first question is "is the system healthy", and the answer is a dashboard of metrics. The second question is "what went wrong", and the answer is structured logs with context. The third question is "where in the distributed path did it fail", and the answer is traces.

The teams that start with a tool, an expensive observability platform with a hundred features, and then wonder why the 3am hour is still painful, have the order backwards. The tool is the vehicle. The questions are the journey. Start with the questions, and the 3am hour gets shorter every time.

The pager will go off. It always does. The only question is whether the next hour is spent looking at a dashboard, following a trace, and finding the root cause, or reconstructing what happened from a vague error message and a hope. The answer is decided long before the pager, in the boring work of wiring the metrics, writing the logs with context, and propagating the trace IDs. That is the whole job. Do it in that order.