Pipeline and quality gates

What to run on a PR, on merge and nightly

The main idea of an automated-test pipeline: the more expensive a suite is, the less often and the later it runs. A single "full regression on every commit" is the shortest road to a 40-minute wait and developers who stop looking at CI.

Pull request (goal: feedback within ~10 minutes): lint and typecheck, unit, API/contract tests, build, an e2e smoke set on critical paths (login, the main product action). All of it are required checks: merge is blocked without green;
Merge to main: a wider set (full API, e2e for the main modules), deploy to staging and a post-deploy smoke. A failure here means the PR set missed something, which is a reason to extend it;
Nightly / scheduled: full regression, all browsers, long scenarios, visual tests, performance checks, a run with quarantine disabled. The result is a report and a team notification, not a merge block;
Manual run (workflow_dispatch): repeat a long suite on demand, for example on a given branch or environment.

The order of stages within a run is fail-fast: cheap and quick checks (lint, unit) go first, slow e2e after them. Independent stages run in parallel, not one after another.

A quality gate is a set of conditions without which code does not move on: all required checks are green, no new critical errors, coverage of new code is not below the threshold, flaky tests are not in the blocking set. The fewer gates "on trust", the better: a condition must be checked automatically.

Flaky tests in CI: retries, quarantine, re-runs

A flaky test is one that sometimes passes and sometimes fails on the same code. In CI it costs twice: a red PR for no reason and a loss of trust in the suite. So you need a policy, not "let's just re-run".

Retries. Allow 1–2 retries in CI (in Playwright retries: process.env.CI ? 2 : 0) and 0 locally, so a developer sees instability immediately. A retry is not a cure but a sensor: a test that passed only on a retry gets the flaky status, lands in the metrics and leads to a ticket. Turning on trace for retries (on-first-retry) is convenient.

Quarantine. A test that is unstable and blocks others is moved out of the required set (a tag, a separate project, a skip with a ticket link) but not deleted: it has an owner, a reason and a deadline. The quarantine set keeps running nightly to see whether it has "recovered".

Re-running. If infrastructure failed (a runner, the network, a rate limit), GitHub Actions has Re-run failed jobs: only the failed job (for example one shard) restarts, not the whole pipeline. It saves time but does not replace root-cause analysis.

What not to do: set a global retries: 5 (it hides real race-condition bugs), automatically re-run red runs without recording it in metrics, leave a skip without an owner.

How to keep PR feedback within 10 minutes

After ~10 minutes a developer switches to another task, and returning to a red PR costs context. When a pipeline is slow, you measure first rather than optimising at random: the time breakdown by jobs and steps (checkout, install, build, tests), p50/p90 duration, queue time for a runner.

The levers, from cheap to expensive:

Parallel jobs. lint, unit, build, API and e2e need not wait for each other when there is no dependency between them;
Cache. The npm cache, Playwright browsers, Docker layers, build output;
Sharding e2e. --shard=i/N in a matrix; the duration is about the slowest shard, so evenness matters;
Selective runs. Path filters and affected-package analysis in a monorepo: a PR that only changes docs or one service does not run everything. The full set stays on main and nightly as a safety net;
The pyramid. Move checks from e2e to API/component tests; e2e on a PR is smoke only;
Stability. Retries and re-runs also eat time: a low flaky rate speeds the pipeline up;
Bigger runners. The last option: usually the bottleneck is not the CPU.

The risk of selective runs is missing a dependency. So a fallback is needed: a periodic full run and a rule "a change to a shared library runs everything".