DAU, MAU, conversion, funnel, retention, churn
Product metrics answer the question "do people use us and do they get value". QA does not have to calculate them every day but should understand what they mean, because they reveal broken places that no test sees.
- DAU / MAU — unique active users per day / per month. The DAU/MAU ratio (stickiness) shows what share of the monthly audience returns daily. 0.2 means a user visits about 6 days out of 30;
- Conversion — the share of those who completed a target action among those who could: a purchase out of visits, a sign-up out of page views;
- Funnel — a sequence of steps with the conversion between them: product view → add to cart → checkout → payment. A funnel shows at which step people drop off. A sharp drop at one step after a release is a reason to check exactly that step;
- Retention — the share of users who came back N days after the first visit (D1, D7, D30). It is measured by cohorts, i.e. by the date of the first visit;
- Churn — the share of those who left during a period (for subscriptions, cancelled). Roughly the "mirror" of retention.
A north star metric is one main metric that best reflects value to the user (for example, weekly orders or minutes of music played). It is complemented by guardrail metrics (speed, errors, complaints) so that growth of the main one does not come at the cost of quality.
A trap: vanity metrics — numbers that look nice but are not tied to value (total sign-ups, clicks). A metric is useful when a decision depends on it.
How to test analytics events and the tracking plan
Analytics is the data on which a team decides what to develop, what to switch off and whether an A/B test won. A broken event produces no errors and does not fail a test, but it poisons decisions. So it is a real bug and deserves checking as seriously as functionality.
A tracking plan is a document (a table, a schema) describing every event: its name, when it fires, required and optional properties with types, platforms. It is the "specification" for analytics testing. Without it you cannot say whether an event is sent correctly.
What to check:
- Firing: the event goes exactly when it should (clicking "Pay" is not a successful payment) and does not go when it should not;
- Name: matches the plan and the naming style (
checkout_completed, not CheckoutDone);
- Properties: all required ones are present, with correct types (an amount is a number, not a string), currency, product id, source. Empty values and
null are defects too;
- Duplicates: the event is not sent twice on reloading the success page, a double click, a request retry, going "back";
- Identification: after sign-in, events are tied to the user, and an anonymous session links correctly to the account;
- Consent: if a user declined tracking, the restricted events are not sent (before and after declining), and changing the decision updates the behaviour. This is not only data quality but also a legal requirement;
- Platforms: web, iOS and Android send the same names and properties.
How to look: the Network tab in DevTools (requests to the analytics service), the debug mode of the tool itself (GA4 DebugView, Mixpanel/Amplitude live view), a proxy such as Charles. In automated tests you can intercept requests and compare them with the expected schema.