Test framework architecture

Test framework layers

A scalable framework is not a folder with tests but several layers with clear boundaries. Each layer knows only about the layer beneath it.

tests/            - scenarios: what we check (Given/When/Then)
steps/ or flows/  - business steps: createUser, makeTransfer
pages/ + api/     - Page Objects and typed API clients
fixtures/         - lifecycle: login, data, clients
config/           - environments, secrets, timeouts
reporting/        - reporters, artifacts, notifications

Why layers. A test must read like a business scenario: "a user with a balance of 100 transfers 30". The details of locators and HTTP requests are hidden below. When the UI changes, you fix one Page Object, not 200 tests.

Separately about API clients. In a fintech project most preconditions must be created through the API, not the UI: create a user, pass KYC, top up an account. This is many times faster and more stable. We leave the UI only for what the test itself verifies.

Fixtures in Playwright are the main composition mechanism. With test.extend you describe what a test receives ready: an authorised page, an API client, a user with a balance. A fixture itself cleans up after the test.

type Fixtures = { api: BankApi; funded: User };

export const test = base.extend<Fixtures>({
  api: async ({ request }, use) => {
    await use(new BankApi(request));
  },
  funded: async ({ api }, use) => {
    const user = await api.createUser();
    await api.deposit(user.accountId, 100);
    await use(user);
    await api.deleteUser(user.id);
  },
});

Task: 2000 e2e tests must finish in 15 minutes

A classic task. We break it down by the scheme.

Requirements and calculation

We clarify: an average test of 20 seconds, a target of 15 minutes for the whole suite, a run on every merge into main. We calculate: 2000 x 20 s = 40,000 s, which is about 667 minutes sequentially. We leave 3 minutes for starting containers, installing browsers and merging reports, so the tests themselves get 12 minutes. 667 / 12 = about 56 parallel slots. We take it with a margin: 8 machines (shards) of 8 workers each = 64 slots.

The solution

• Test isolation: each test creates its own data, with no dependencies between tests. Otherwise parallelism is impossible.
• Parallelism: fullyParallel: true, sharding through --shard=i/8, reports in blob format and merging through merge-reports.
• Preconditions through the API, login through a saved storageState.
• A ready image with browsers and dependencies, so as not to spend minutes on installation.
• Reviewing the pyramid: some of the 2000 e2e tests can be moved to the API level. That often gives more than new machines.

Trade-offs

More machines mean higher CI cost and more load on the test environment. 64 parallel sessions may "bring down" staging, so the environment must be scaled too, or each shard given its own.

Metrics

Pipeline time p50 and p95, flaky rate, shard imbalance (the longest shard against the average), the cost of one run.

Configuration, environments and secrets

One framework must run locally, on a PR environment and on staging without changing code. So config is a separate layer.

Principles:

• All environment variables are read in one place and validated at startup. If BASE_URL isn't set, the run fails immediately, not on the 300th test.
• Secrets (API keys, test users' passwords) only from the CI secret store or a vault, never in the repository.
• Timeouts are set centrally in playwright.config.ts, not scattered across tests.
• Projects in Playwright separate runs: api, chromium, mobile, smoke. You can run separate sets through --project.

export default defineConfig({
  fullyParallel: true,
  retries: process.env.CI ? 1 : 0,
  workers: process.env.CI ? 8 : undefined,
  reporter: process.env.CI ? 'blob' : 'html',
  use: {
    baseURL: env.BASE_URL,
    trace: 'on-first-retry',
  },
  projects: [
    { name: 'setup', testMatch: /.*\.setup\.ts/ },
    { name: 'chromium', dependencies: ['setup'] },
  ],
});

The setup project with a dependency is the standard way to log in once and save a storageState that all tests then use.

A separate question is several teams. If 5 teams write tests, the shared core (clients, fixtures, config) is worth moving into an internal package with versioning, and the tests kept next to the teams' services. Then the core evolves centrally, and the teams don't block each other.