CI/CD, parallelization and sharding

Tests in the pipeline: stages and quality gates

A good pipeline gives fast feedback with cheap tests and runs expensive tests when it makes sense.

PR:      lint + typecheck -> unit -> API -> e2e smoke (5-10 min)
main:    full API + full e2e (up to 15 min)
nightly: cross-browser, long scenarios, performance
release: smoke on staging -> deploy -> smoke in production

A quality gate is a rule that blocks a merge or a deploy. Examples: all unit and API tests are green, the e2e smoke is green, coverage of changed files didn't drop. Important: a gate must be reliable. If the smoke is flaky, teams will start ignoring it, and the gate loses its meaning.

What to run on a PR. The full suite on every PR is often too expensive. Options:

• A smoke set tagged @smoke through --grep.
• Tests related to changed files: Playwright has --only-changed, which takes changes relative to a git branch. It works for tests that import the changed modules, so for a backend change it won't help.
• Rerunning only the failed: --last-failed locally.

For fintech I always include the critical money scenarios in the PR set: a transfer, a top-up, a refusal on insufficient balance. Their cost of error is the highest.

Workers and shards: how parallelisation works

Playwright has two levels of parallelism.

• Workers are processes on one machine. Each worker has its own browser. The number is limited by the machine's CPU and memory.
• Shards split the test suite between several machines through --shard=1/4, --shard=2/4 and so on.

strategy:
  matrix:
    shard: [1, 2, 3, 4, 5, 6, 7, 8]
steps:
  - run: npx playwright test --shard=${{ matrix.shard }}/8
  - uses: actions/upload-artifact@v4
    with:
      name: blob-${{ matrix.shard }}
      path: blob-report

After all the shards a separate job downloads all the blob reports and calls npx playwright merge-reports --reporter html ./all-blob-reports. That gives you one HTML report.

Important nuances:

• By default Playwright parallelises files, and tests inside a file run sequentially. With fullyParallel: true individual tests are parallelised, and the shards are balanced better.
• If one file has 200 tests and the others have 5 each, without fullyParallel one shard will be much longer.
• The number of workers on a machine must not exceed its resources. 8 browsers on 2 vCPUs will give timeouts and flakiness.
• Tests with mode: 'serial' aren't parallelised inside the group. They should be minimised.

The efficiency metric: the difference between the longest and the average shard. If the longest shard is 14 minutes and the average is 6, you pay for idle time.

Feedback speed and cost

A Senior must talk not only about speed but also about money and where the real bottleneck is.

Where pipeline time comes from:

• Runner start and dependency installation. The solution: caching node_modules, a ready Docker image with browsers.
• Data preparation and login. The solution: the API instead of the UI, storageState.
• Slow tests. The solution: find the top 20 longest and rewrite them or move them to the API.
• Waiting for the environment. The solution: a health check before starting, not sleep.

When to add machines and when not. If tests run into the shared staging (the DB, a rate limit), adding shards won't speed up the run but will increase timeouts. First you must find the bottleneck.

Another optimisation is fail fast. The maxFailures option stops the run after N failures. On a PR this saves minutes when something basic is broken, for example login.

Cost. Roughly calculate: 8 machines for 15 minutes on every merge, 50 merges a day. That is already noticeable money. So nightly for rare scenarios and smoke on a PR is not laziness but economy.