Quality metrics

what to measure, how to count, what to avoid

How do you choose quality metrics for a project?

Not from a list of "popular metrics", but from a goal. A convenient framework is GQM (Goal → Question → Metric):

Goal: "reduce the number of defects that reach production".
Questions: how many defects do we miss? At which stage should we have caught them? Which modules are the most problematic?
Metrics: defect leakage, the distribution of production bugs by module and by the stage where they could have been found.

Rules: every metric has an owner and an action it triggers; there are few metrics (3–5 key ones); we look at the trend, not a single value.

In an interview: I start with the goal, not the metric, following the Goal-Question-Metric approach. For example, the goal is fewer bugs in production, the question is how many we miss and where we should have caught them, and the metrics are defect leakage and the distribution of production bugs by module. I keep three to five key metrics, each of which leads to a concrete action, and I look at trends, not individual numbers.

What are defect leakage, DRE and defect density and how do you calculate them?

Metric	Formula	What it shows
Defect leakage	production bugs ÷ (bugs before release + production bugs) × 100%	What share of defects we miss
DRE (Defect Removal Efficiency)	bugs before release ÷ (bugs before release + production bugs) × 100%	Catching efficiency: 100% − leakage
Defect density	number of defects ÷ size (module, KLOC, story points)	Which parts are the most problematic

Example: in a release 45 bugs were found before production and 5 after. Leakage = 5 ÷ 50 = 10%, DRE = 90%.

A nuance worth saying: count only significant defects (for example Major and above) and over a fixed window after the release, otherwise minor things distort the picture.

In an interview: Defect leakage is the share of defects we missed into production out of all those found. DRE is the opposite, the share found before release. For example, forty-five bugs before release and five after is ten percent leakage and ninety DRE. Defect density shows which modules are the most problematic. I count only significant defects and over a fixed window after release, so that minor things don't distort the picture.

How does requirements coverage differ from code coverage? Is 100% needed?

Requirements coverage is what share of requirements or acceptance criteria have tests, and how many of them passed. It is calculated through a traceability matrix or links in Jira/Zephyr.

Code coverage is what share of lines or branches is executed by tests. Measured by tools, mostly for unit tests.

100% is not the goal. High code coverage doesn't guarantee quality checks (a test may execute a line and verify nothing), and 100% of requirements may mean superficial cases. More important is risk coverage: critical scenarios covered deeply, secondary ones sufficiently.

In an interview: Requirements coverage shows what share of requirements has tests, and code coverage what share of the code is executed by tests. A hundred percent isn't a goal for me: a test may execute a line and verify nothing. I aim at risk coverage: critical scenarios must be covered deeply, and secondary ones sufficiently.

Which process metrics do you track?

Defect fix time by severity: do we meet the agreed deadlines.
Reopen rate: the share of bugs that get reopened: a signal about the quality of fixes or retesting.
Rejected rate: the share of bugs rejected as "not a bug" or a duplicate: a signal about the quality of reports or requirements.
Test execution progress: executed, passed, blocked as of the current release moment.
Time in the "Ready for QA" status: whether testing is becoming a bottleneck.

Each of them is a reason for a conversation at the retro, not an evaluation of people.

In an interview: I look at defect fix time by severity, reopen rate, how many bugs get reopened, rejected rate, how many are rejected as not a bug or a duplicate, test execution progress and how many tasks are waiting for QA. A high rejected rate, for example, hints that the problem is in the requirements or in my reports, and that is a topic for the retro.

Which metrics do you use for automation?

Automation coverage: the share of regression cases that are automated, especially critical ones.
Flaky rate: the share of tests that sometimes fail and sometimes pass with no code changes. The goal is close to zero, because flakiness destroys trust in the whole suite.
Run time and its trend: whether feedback on a PR is slowing down.
Pass rate on main: stability.
Defects found by automated tests: whether automation really catches regressions.
ROI: how many hours of manual regression are replaced minus maintenance costs.

In an interview: For automation I track the share of automated regression, flaky rate, run time, the pass rate on the main branch and how many regressions the automated tests actually caught. Flaky rate is the most important for me: if the team stops trusting red tests, automation loses its point.

Which metrics are harmful and why?

Goodhart's law: when a measure becomes a target, it ceases to be a good measure.

The number of bugs found as a KPI → minor bugs get split into several, conflicts with developers, and nobody works on preventing defects.
The number of test cases or automated tests written → many small and duplicated cases.
"Zero bugs in production" as QA's personal goal → fear of releases and excess bureaucracy.
Metrics for comparing people → distorted data.

Metrics must show the state of the system and the process, not evaluate individual people.

In an interview: I am careful with metrics like the number of bugs found or cases written as a KPI. By Goodhart's law, when a measure becomes a target, people optimise the number rather than the quality: bugs get split, cases get duplicated. Metrics must show the state of the product and the process, not evaluate people.

How do you build a quality dashboard for different stakeholders?

Different people need different answers:

Business / PM: is the release ready, open blockers and critical issues, risks, the trend of production incidents.
The team: sprint testing progress, bugs by status, flaky tests, time in "Ready for QA".
Management: trends over a quarter: leakage, fix time, release stability.

Technically: JQL filters in Jira and gadgets on the dashboard, Zephyr or Xray reports, Allure for automated tests; for more complex needs, Grafana or BI tools.

-- open critical bugs of the current release
project = PAY AND issuetype = Bug AND fixVersion = "2.15"
  AND priority in (Highest, High) AND statusCategory != Done

-- production bugs for the last 30 days (for leakage)
project = PAY AND issuetype = Bug AND labels = production
  AND created >= -30d ORDER BY priority DESC

In an interview: I build a dashboard for the audience. The PM needs an answer on whether the release is ready and what the blockers are, the team needs sprint progress and bug statuses, management needs quarterly trends. Technically these are JQL filters and gadgets in Jira, Zephyr reports and Allure for automated tests.

How do you prove quality improved? What are DORA metrics?

You need a baseline (the state before changes), a trend over several releases and a link to a change in the process. One number proves nothing.

DORA metrics are four delivery performance metrics that assess both speed and stability:

Deployment frequency: how often we release.
Lead time for changes: from commit to production.
Change failure rate: the share of releases that caused an incident or a rollback. The closest to QA.
Time to restore: how long recovery after an incident takes.

A strong answer: quality improved if change failure rate and leakage fall, while release frequency doesn't, that is, we didn't buy stability at the price of speed.

In an interview: I record a baseline and show a trend over several releases, tied to a specific process change. DORA metrics suit this well: deployment frequency, time from commit to production, the share of failed releases and recovery time. Real improvement is when the share of failed releases falls and release frequency doesn't.