Reporting, metrics, observability

A report that helps find the cause

A report must answer three questions: what failed, why and whether it failed before. A list of green and red tests is the minimum, not the goal.

What a report must have for every failure:

The assertion message and the stack.
The Playwright trace: steps, DOM snapshots, network requests, console.
A screenshot and video (as needed, because they take space).
The requests' correlation ids, to find backend logs.
The environment, build version, shard number.

Reporting levels:

The Playwright HTML report is for an engineer analysing a failure.
History and trends (Allure or your own results DB) are for finding flaky and slow tests.
A chat notification: briefly, how many failed, a link to the report, who authored the commit.
A dashboard for the team and management: stability, pipeline time, the trend.

A mistake I often see: a notification for every failure in a general channel. A week later everyone ignores the channel. It is better to send it to the PR author or the area's owner, and only failures on main to the general channel.

Separately keep raw results (JSON or JUnit XML). Then you can build your own analytics without being tied to one reporting tool.

In an interview: I set up a report so that it answers what failed, why and whether it failed before: a trace, a correlation id, the test's history. Notifications go specifically to the change's author, because a general noisy channel quickly stops being read.

Test system metrics

What isn't measured isn't improved. Here are the metrics I would show on a dashboard.

Metric	What it shows	Benchmark
Pipeline time p50 and p95	Feedback speed	A PR under 10-15 min
Flaky rate	The share of tests that passed only with a retry	Under 1-2 percent
Pass rate on main	The health of the main branch	Close to 100 percent
Escaped defects	Bugs found in production	A downward trend
Failure analysis time	How much time goes into analysis	A downward trend
Run cost	Money on CI	Under control

Why p95 and not the average. An average time of 10 minutes may hide that every fifth run takes 25 minutes because of a slow shard. Developers feel exactly the tail.

Escaped defects are the most honest quality metric. Analyse every production bug: could a test have caught it, at which level, why it didn't. That gives concrete actions, not an abstract "increase coverage".

Careful with code coverage. High coverage doesn't guarantee quality checks. A test without meaningful assertions also gives coverage. For e2e, coverage of business scenarios or risks is more useful: are all payment methods, currencies and transfer statuses covered.

In an interview: I look at the p95 of pipeline time, flaky rate, pass rate on main and escaped defects. What gives me the most is analysing production bugs: for each I ask at which level a test should have caught it, and I turn exactly that into a plan.

Observability and testing

Observability is the ability to understand a system's state from its external signals: logs, metrics, traces. For QA it is a way to find the cause of a failure in a distributed system, not only in the browser.

A correlation id (or trace id) is an identifier that passes through all the services within one request. A test can pass its own header and then use it to find logs in all services.

test('transfer completes', async ({ request }, testInfo) => {
  const corrId = `e2e-${testInfo.testId}-${Date.now()}`;
  const res = await request.post('/api/transfers', {
    headers: { 'X-Correlation-Id': corrId },
    data: { from: 'acc_1', to: 'acc_2', amount: 30 },
  });
  testInfo.annotations.push({ type: 'correlationId', description: corrId });
  expect(res.status()).toBe(201);
});

The header's name depends on the project: it may be X-Request-Id or the standard traceparent from W3C Trace Context.

How else QA uses observability:

Checks not only the API response but also that an event reached the queue and the audit log.
After a deploy looks at error and latency metrics as part of the release check.
Synthetic monitoring is a small set of scenarios that regularly go to production (for example login and viewing the balance on a test account) and alert on failure.

In an interview: I pass a correlation id from the test and store it in the report, to find the logs of all services for a specific failure within a minute. After a release I use metrics and synthetic checks in production as a continuation of testing.