Observability

Logs, metrics, and traces: what each is for

Three signals, three jobs. Metrics are cheap aggregate numbers over time (request rate, error rate, memory used); they answer "how many" and "how fast" and drive dashboards and alerts. Logs are timestamped events with detail; they answer "what exactly happened in this one request." A trace follows a single request across every service it touches and shows where the time went. You reach for metrics to notice a problem, traces to locate it, and logs to read the specifics once you know where to look.

The trap is treating them as interchangeable. Metrics cannot tell you which user hit the bug. Logs cannot cheaply tell you the p99 across a million requests. You want all three, joined.

Which metric type to use

Four types exist; three you reach for daily, and picking wrong bites you later:

• Counter: monotonically increasing, resets to 0 on restart. Totals like http_requests_total. You graph rate() of it, never the raw value.
• Gauge: goes up and down. In-flight requests, queue depth, memory in use.
• Histogram: buckets observations (request duration, payload size) so you can compute percentiles later, and it aggregates across instances.

For dashboards, two recipes cover most services. RED per service: Rate, Errors, Duration. USE per resource: Utilization, Saturation, Errors. These map onto Google's four golden signals (latency, traffic, errors, saturation), which is the vocabulary an interviewer expects when they ask "what would you put on the dashboard?"

How correlation ids join the pillars

A log line nobody can query is close to useless. Log structured (key-value or JSON), and stamp every line with the ids that let you pivot between signals:

{"level":"error","msg":"charge failed","trace_id":"a1b2c3","user_id":42,"latency_ms":812}

Wrong "we have logs, so we have observability." Unstructured print lines scattered across ten services give you grep and despair. The thing that turns three separate signals into a debugging tool is a shared trace_id: your metrics show error rate spiking, you click into a slow trace, you grab its trace_id, and you pull every log line from every service for that exact request. Without that id you are correlating by wall-clock timestamp and guessing.

SLIs, SLOs, and error budgets

An SLI is a measured indicator of health, usually a ratio of good events to total ("fraction of checkout requests served under 300 ms"). An SLO is the target you hold it to ("99.9% over 28 days"). The error budget is the leftover: 99.9% means 0.1% of requests may fail before the budget is spent. Budget framing lets a team say no to risky launches when reliability is underwater and yes when there is room. The number depends on the window, so agree on the window before arguing about the target.

How distributed tracing works

A trace is a tree of spans. Each span records a start, a duration, a name, attributes, its own span id, and the trace id shared across the request; a child also carries its parent's id, which is how the collector rebuilds the tree. The hard part is joining spans across process boundaries. When service A calls service B, A serializes the trace context into the request (the W3C traceparent header), and B makes its spans children of A's:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             ^  ^--------- trace-id ---------^  ^--- span-id --^ ^flags

Drop that header at any hop (a queue, a thread pool, an async job that loses the context) and the trace splits into two disconnected halves. Most "why is my trace incomplete" bugs are a propagation gap, not a sampling one.

Sampling is the other senior topic. Head-based sampling decides at the root span before the request runs, so it is cheap but blind: it cannot keep the slow or errored traces because it has not seen them yet. Tail-based sampling buffers the spans and decides after the trace completes, so it can retain every error and every slow request, at the cost of holding spans in a collector. Large systems run tail sampling for that reason.

Metric cardinality and the histogram vs summary choice

Cardinality is the product of distinct label-value combinations, and each combination is a separately stored time series. Prometheus holds every active series in memory, so one unbounded label detonates the whole backend.

Wrong "add user_id as a label so we can slice per user." That is one series per user; at a million users you OOM the target and the server. Per-request identity belongs in traces and logs, or in a metric exemplar that links a bucket to a sample trace. Metric labels must be bounded: method, status code, a templated route (/orders/:id, never the raw path).

The histogram vs summary decision is a real correctness trap. A summary computes its quantiles client-side inside each instance, so you cannot aggregate them: averaging a per-instance p99 across ten replicas is meaningless, since percentiles are not linear (the percentile discipline lives in the backend-performance module). A histogram ships raw buckets that the server aggregates across instances before computing the quantile. Run more than one replica and want a fleet-wide percentile, and you need a histogram.

Error budgets and burn-rate alerting

The error budget is 1 minus the SLO over a rolling window. At 99.9% over 30 days that is about 43 minutes of allowed failure; over a 28-day window it is about 40. Burn rate is how fast you are spending it: a burn rate of 1 exhausts the budget exactly at the window's end, and a burn rate of 14.4 held for an hour spends 2% of a 30-day budget (Google's fast-burn page threshold). Exhausting a 30-day budget in a single day takes a sustained burn rate near 30.

Good alerting is multi-window, multi-burn-rate. A fast burn (budget gone in hours) pages immediately; a slow burn (gone in days) opens a ticket. Requiring a short and a long window to both fire before paging kills the false alarms from a brief blip. This is the concrete form of "alert on symptoms, not causes": the page fires because users are losing requests against an SLO, not because a CPU gauge crossed a round number.

How to debug a production incident, and what gets graded

The graded answer is a funnel. Scope with metrics: which endpoint, which SLI, since when, off the golden-signal dashboards. Localize with a trace: pull one slow or errored request and read where the wall time and the failure sit, which service, which downstream call. Confirm with logs joined by trace_id, then apply the smallest fix and check that the SLI recovered.

Interviewers listen for whether you reach for a trace before guessing a cause, whether you know which signal answers which question, and whether per-request identity goes in logs rather than metric labels. Candidates who propose a code rewrite before they have looked at a single trace are the ones who miss.