A/B tests and feature flags

A/B tests from the QA point of view

An A/B test is an experiment in which the audience is randomly split into groups: control (the current experience) and one or more variants (the change). Then the target metric (conversion, retention) is compared and statistics decide whether the difference is not random. It answers "does the change really help", not "do we like it".

What QA should know:

• Assignment stickiness. A user must always land in the same group: after a reload, signing in from another device (if there is an identifier), a new session. If a person sees A and then B, their data distorts the result and the experience looks like a bug;
• Check every variant. Each variant is a separate version of the product. Functionality, layout, analytics, localization and accessibility need tests in each. A common mistake is to test only "the new one";
• Forced assignment. For testing you need a way to land in a variant (a query parameter, a cookie, an internal tool) rather than wait for chance. Such a mechanism must not be open to everyone in production;
• Experiment analytics. The exposure event (the user saw a variant) must be sent once and with the group name. Otherwise the metrics will compare the wrong people;
• Do not "peek" at the result. If you stop the test as soon as a variant pulls ahead, there is a high chance of a false "win" caused by random fluctuations. The duration and the required sample size are set in advance, and QA does not push for an early conclusion;
• Overlapping experiments. Two tests on one screen may influence each other. They are separated (mutually exclusive groups, layers) or accounted for in the analysis;
• Quality of variants. A variant that breaks some browsers or slows the page loses because of a defect, not because of the idea. QA protects the purity of the experiment.

"Sample size" is enough to know conceptually: the smaller the expected difference and the higher the required confidence, the more users and time are needed; a few days on low traffic usually prove nothing.

Feature flags: combinations, kill switch, cleanup

A feature flag is a switch in configuration that turns functionality on or off without a new deployment. They are managed by custom configs or services such as LaunchDarkly or Unleash. Flags allow merging unfinished code, opening a feature gradually, enabling it for part of the users and switching it off quickly when something goes wrong.

How to test a flag-guarded feature.

• Both states. Flag off — the behaviour is as before (no regression); on — the new one. The first is often forgotten;
• The transition. What happens when the flag is turned on and a user already has data created by the old logic? What when it is turned off in the middle of a session?
• Targeting. Rules (percentage, country, role, app version) work as intended, including edges and unauthenticated users;
• Combinations. n flags give 2ⁿ states. You cannot check them all, so take those that really coexist in production and pairs that touch the same area of code;
• Environments. A flag is configured differently in dev, staging and prod. Before a release, check its production value, not just the test one.

A kill switch is a flag meant for emergency shutdown of a risky function (payment through a new provider, a heavy query). It is checked in advance: switching off really takes effect quickly, without a restart; the app behaves gracefully (a fallback or a clear message), does not crash and does not lose the data created while the feature was on.

Cleanup. Every flag is temporary complexity. A flag that has long been on for everyone leaves a dead code branch and tests for it, and a trap: someone will switch it off "to try". So when a flag is created, an owner and a review date are assigned; after the full rollout the old branch and its tests are deleted. Long-forgotten flags are technical debt that QA should remind about.