QA Interview Hub · Knowledge base · Quizzes

Logs and observability

Observability: logs, metrics, traces

Logs, metrics and traces: three signals

Monitoring answers questions we anticipated ("is CPU above 90%?"). Observability is the ability to explain the system's state from its external signals, including questions nobody anticipated ("why is checkout slow only for Safari users in Frankfurt?"). The main signals:

  • Logs — individual events with context. They answer "what exactly happened in this request". Costly in volume, the most detailed.
  • Metrics — numeric time series (requests/s, error rate, p95 latency, memory use). Cheap, easy to aggregate, show what and how much but not why. The basis of dashboards and alerts.
  • Traces — the path of one request through services, split into spans with durations. They answer "where exactly time was lost or an error occurred".

A typical investigation: a metric alert (the 5xx rate grew) → a trace shows that a slow/failing call is in the payments service → the logs of that span, found by trace id, explain the cause (a gateway timeout).

Useful metric sets: RED for services (Rate, Errors, Duration) and USE for resources (Utilization, Saturation, Errors). There are also the golden signals: latency, traffic, errors, saturation.

For a QA: besides functional checking of a response, a test or a post-release manual check looks at errors and latency in the metrics; synthetic monitoring tests regularly run key scenarios in production.

Distributed tracing, trace ids and OpenTelemetry

In microservices a single user click produces a chain of calls. Distributed tracing links them into one trace: the root is the whole request and a span is a single operation (an HTTP call, a database query) with a start, a duration, attributes and a reference to its parent span.

For services to know they belong to one trace, the context is passed in headers. The standard is W3C Trace Context, the traceparent header:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             version-trace id (32 hex)-parent span id (16 hex)-flags

The flag 01 means "sampled" (the trace is recorded). The extra tracestate header carries vendor data. Older systems use their own X-Request-Id, X-Correlation-Id or B3; this identifier must be present in every log entry, so that one id finds both the trace and all related logs.

OpenTelemetry (OTel) is an open, vendor-neutral standard and set of SDKs for logs, metrics and traces:

  • API/SDK for languages (manual and automatic instrumentation — HTTP, database and framework libraries);
  • OTLP — the wire protocol;
  • Collector — an intermediate service that receives, processes (filtering, masking, sampling) and forwards data to a backend (Jaeger, Tempo, Datadog and others).

Sampling: recording every trace is expensive. Head sampling decides at the start (for example 5% of requests), tail sampling decides after completion and can therefore keep all failed and slow traces.

What it gives a QA: a test or a manual check can take the trace id from the response (or generate a traceparent itself) and, after a failure, open the full request path and see which service broke. This is especially valuable in e2e, where the UI only shows "something went wrong".

Dashboards, alerts and SLI/SLO for a QA

A dashboard shows the state of the system on one screen: error rate, p95 latency, traffic, queues. A good dashboard answers "is everything fine?" in seconds and leads to details (drill-down). An alert is an automatic notification when a metric violates a condition.

Alerting principles:

  • alert on user-facing symptoms (error rate, latency), not on every internal detail;
  • every alert must be actionable: someone knows what to do (a runbook); noisy alerts get ignored ("alert fatigue");
  • averages hide problems — look at percentiles (p95/p99), not only the mean.

SLI / SLO / SLA in short:

  • SLI (indicator) — a measurable quality figure, for example the share of successful requests or the share of responses faster than 300 ms;
  • SLO (objective) — a target for an SLI over a period: "99.9% of requests succeed over 30 days";
  • SLA — an external contractual promise with consequences for a breach;
  • Error budget — the allowed share of failures (100% − SLO; for 99.9% it is 0.1%). While the budget is not spent, the team can ship changes quickly; once it is spent, reliability takes priority.

Where a QA fits: checks that key scenarios are covered by metrics and alerts (for example, that an alert really fires when an error is injected in a test environment); uses SLOs as a release criterion (canary release, rollback by metrics); adds synthetic tests; analyses whether a test could have caught an incident and adds it.