CI reports, artifacts and metrics

Reports and artifacts: what to keep and how to share

Which reports. The Playwright HTML report (playwright-report/) is for people: steps, screenshots, trace. JUnit XML is for machines: CI and third-party services show tests in their UI, draw history and annotate PRs. Usually both are enabled: reporter: [['html', { open: 'never' }], ['junit', { outputFile: 'results/junit.xml' }]], and in CI you add github (annotations in the logs) or dot (short output).

What to upload. Traces, video and screenshots are the best evidence of a failure but heavy. Record them only for failed tests or retries (trace: 'on-first-retry', video: 'retain-on-failure', screenshot: 'only-on-failure'). Set retention-days for artifacts (7–14 days is enough) and do not upload what is not needed on success.

Sharded runs. Each shard produces a blob report; a separate job (needs: [e2e]) downloads all blob artifacts (download-artifact@v4 with pattern and merge-multiple: true), runs npx playwright merge-reports --reporter html ./all-blob-reports and uploads a single HTML report. The merge job also needs if: ${{ !cancelled() }}, otherwise it will not start after a red shard.

How to share. A zip artifact has to be downloaded and unpacked, and an HTML report with static assets cannot be opened as a single file without a server. It is more convenient to publish the report to GitHub Pages or static hosting and add the link to a PR comment. For a short summary on the run page, write Markdown to the $GITHUB_STEP_SUMMARY file.

Notifications. Send only what matters to a channel (Slack/Teams): failures of nightly and main, not of every PR; the message has the suite name, the number of failures, and links to the run and the report.

CI metrics and failure triage

A pipeline is a product that also needs measuring. The minimal set:

• Pipeline duration: p50 and p90/p95, not just the average. The average hides the long tail, which is what annoys the team. Separately, the time a job waits for a runner in the queue;
• Flaky rate: the share of tests (or runs) that passed only after a retry, out of all of them in a period. Watch the weekly trend and the top most unstable tests;
• Pass rate of the main branch and the share of red PRs caused by non-code reasons (infrastructure, flaky tests);
• MTTR of a red main: how fast green is restored;
• Cost: runner minutes per PR, artifact size.

Failure triage (for example, nightly: 30 red tests). Do not open them one by one:

1. Group by error message and stack: usually 30 failures are 2–3 causes;
2. Find out when it started: the last green run and what changed between them (commits, a deploy, a dependency version, data);
3. Classify: a product defect, a test defect (locator, race, data), infrastructure (timeout, runner, external service);
4. A product defect gets a ticket to developers with a trace; a test defect gets a fix or quarantine with an owner; infrastructure gets a re-run and a ticket to the platform team;
5. Close the loop: add a check that prevents recurrence.

It is important to tell "red because of a bug" (the test did its job) from "red because of noise". Otherwise the team learns to ignore red.