Stability and flaky tests

Where flaky tests come from

A flaky test is one that, on the same code, sometimes passes and sometimes fails. The main harm is not in one failure but in the team ceasing to trust the results.

The main causes, from the most frequent:

Synchronisation. A test checks the state before the UI or backend updated it. Manual waitForTimeout instead of waiting for a condition.
Shared data. Parallel tests change one account.
Order dependence. A test works only after another test.
An asynchronous backend. A transfer is processed through a queue, and the test checks the balance immediately.
An unstable environment. An overloaded staging, slow external sandboxes, the network.
Time and dates. A test fails at midnight, on the last day of the month or in a different time zone.
Animations and unstable locators. Dependence on CSS classes or the order of elements.

How to tell a product bug from a flaky test. We look at the trace: if the UI really showed a wrong amount, that may be a race condition in the product, and it is a bug, not "flaky". Some flaky tests actually catch real concurrency problems. So "just add a retry" is dangerous.

In Playwright most synchronisation problems are solved by web-first assertions: await expect(locator).toHaveText('70.00') waits and rechecks until the timeout. For the backend there is expect.poll.

In an interview: The most frequent causes of flakiness are synchronisation, shared data, order dependence and an asynchronous backend. Before calling a test flaky, I look at the trace, because sometimes an unstable test catches a real race condition in the product.

The process of fighting flakiness: detect, isolate, fix

One engineer fixing tests doesn't scale. A process is needed.

Detection -> Quarantine -> Analysis -> Fix -> Return

Detection. We collect the history of each test's results between runs. A test that passed after a retry is marked flaky by Playwright. If a test has several such results in a week, it is a candidate. Additionally the suite can be run once a night with --repeat-each=5 to find unstable tests before they bother the teams.

Quarantine. An unstable test is taken out of the blocking gate: the tag @quarantine, a separate non-blocking job. A ticket with an owner and a deadline is always created for it. Without an owner, quarantine turns into a graveyard.

Analysis. Trace, video, backend logs by correlation id. We classify the cause: the test, the data, the environment or the product.

Fix and return. After a fix the test is run many times (for example --repeat-each=20) and only then returned to the gate.

Rules I would set:

A quarantine limit, for example 2 weeks. After that the test is either fixed or deleted.
The flaky rate as a team metric, visible on the dashboard.
A new test is run several times before merge.

In an interview: I build a process: automatic detection from history, quarantine with an owner and a deadline, analysis through trace and logs, and return to the gate only after many stable runs. The key is that every flaky test has an owner, otherwise quarantine grows forever.

Retries: when they help and when they hurt

Retries in Playwright are configured through retries in the config. A test that failed and passed on a repeat gets the flaky status.

When a retry is useful:

As temporary protection against infrastructure failures on CI.
For collecting diagnostics: trace: 'on-first-retry' records a trace exactly when a test failed.
As a source of data for detecting flakiness.

When a retry hurts:

When it hides a product bug. A race condition in a money debit may show up in 1 run out of 10. A retry will make the pipeline green, and the bug goes to production.
When retries are large (3-5). Run time grows, and real stability doesn't improve.
When a step-level retry repeats a non-idempotent action. A repeated transfer POST may create a second transfer.

The difference between a test retry and waiting for a condition. Waiting is a normal part of a test: expect.poll waits until the transfer status becomes COMPLETED. A test retry is restarting the whole test from scratch. The first is correct synchronisation, the second is a backup mechanism.

await expect.poll(async () => {
  const res = await api.getTransfer(id);
  return res.status;
}, { timeout: 15_000, intervals: [500, 1000, 2000] }).toBe('COMPLETED');

In an interview: On CI I set at most 1-2 retries and always track tests that pass only on a repeat. A retry is insurance and a source of diagnostics, not a solution: in fintech it can hide a race condition in debiting money.