Debugging, trace viewer, reports, flaky tests

Diagnostics: trace viewer, reports, flaky tests

The trace viewer is the main debugging tool: a trace records every action, DOM snapshots before and after, the network, the console and the test's source code. On CI trace: 'on-first-retry' is usually set, so as not to spend resources on successful runs. A trace is opened with npx playwright show-trace or from the HTML report.

Locally for debugging there is UI mode (--ui) with time travel, the --debug mode with Playwright Inspector and page.pause(). Codegen (npx playwright codegen) records actions and suggests locators. Reports: the built-in HTML, list, JUnit for CI, as well as Allure.

To look for flakiness you use --repeat-each=10 and --retries, and in the HTML report such tests are marked as flaky. Typical causes: shared data between workers, no web-first assertions, waitForTimeout, animations, a slow CI environment. You treat the root cause: retries only mask the problem.

Local debugging: the tools and when to use which

• npx playwright test --ui is UI mode. A list of tests, watch mode (rerun when a file changes), time travel through steps with DOM snapshots, network and console. My main tool when writing tests.
• --debug or the variable PWDEBUG=1 is Playwright Inspector: step-by-step execution, the browser in headed mode, timeouts disabled. Convenient for stopping before a specific action.
• await page.pause() is a breakpoint in code, which opens the Inspector right here.
• --headed and launchOptions: { slowMo: 300 } let you see what is happening at a slowed-down pace.
• The VS Code extension: running a single test, breakpoints, the Pick locator button, which shows the recommended locator for an element.

Codegen is more useful than it seems, not only for recording actions:

npx playwright codegen --device="iPhone 13" --lang=uk-UA https://app.test
npx playwright codegen --save-storage=auth.json https://app.test/login
npx playwright codegen --load-storage=auth.json https://app.test/dashboard

The first command opens the application with device emulation, the second saves the state after a manual login, the third opens it already logged in. I don't commit the generated code as is: I take the locators from it and rewrite the structure into a Page Object and fixtures.

This set doesn't work for debugging on CI, where the main thing is the trace.

Artifacts and reports

What to record is configured in use:

on-first-retry writes an artifact only on the first retry, so without retries it gives nothing. retain-on-failure always writes, but keeps only for tests that failed: this is more expensive in resources, but the artifact exists even without retries.

You can use several reporters at once:

reporter: process.env.CI
  ? [['blob'], ['junit', { outputFile: 'results/junit.xml' }], ['github']]
  : [['html', { open: 'on-failure' }], ['list']],

github adds error annotations right in the PR, JUnit is read by Jenkins or GitLab, and HTML is for people. For Allure you add allure-playwright.

Your own data is added to the report through testInfo.attach:

test('statement reconciliation', async ({ request }, testInfo) => {
  const res = await request.get('/api/statement');
  await testInfo.attach('statement', {
    body: await res.text(),
    contentType: 'application/json',
  });
});

This way API responses, backend logs and generated files appear in the report. When a test fails, the developer sees not only a screenshot but also the data the test worked with.

How I systematically investigate a flaky test

A flaky test is not "just rerun it". My process:

1. Reproduce. npx playwright test checkout.spec.ts --repeat-each=20. If it fails locally, good, there is something to debug.
2. Check isolation. Run with --workers=1 and separately with --workers=4. If it fails only in parallel, the problem is shared data or an account between workers.
3. Check the order. Run the test alone with -g "name". If it passes alone but fails in the suite, another test depends on it or leftover data.
4. Compare the trace of a successful and a failed run. It often shows that on the failure the API responded slower, or a spinner still covered the button.
5. Classify the cause: the test (no wait, a brittle locator), the data, the environment, or a real application bug, for example a race condition in the front end.

The last point is important: some flaky tests are real bugs that a user will also meet. If a double click creates two payments, it is not a test problem.

Until the cause is fixed, the test must not block the team. I mark it test.fixme with a link to a ticket or move it into a quarantine project with the tag @quarantine, which doesn't block the merge. But quarantine must have an owner and a deadline, otherwise it turns into a dumping ground.

A metric worth tracking is the share of flaky tests in runs. The HTML report marks a test as flaky if it passed only after a retry.