Safe releases and production feedback

Canary, gradual rollout, blue-green, rollback

The main idea of safe releases: limit the blast radius. Instead of "everyone at once", the new version is shown to a small share of traffic, signals are watched, and only then is it widened.

• Canary. The new version runs next to the old one and gets a small percentage of traffic (1-5%). If metrics are fine, the percentage grows; if not, the canary is switched off and only a small share of users suffers. The name comes from the canary in a coal mine;
• Gradual (staged / percentage) rollout. The same at the level of a feature or an app: 1% → 10% → 50% → 100% with pauses to observe. Mobile apps have staged releases in the stores;
• Blue-green. Two full copies of the environment: "blue" serves traffic, the new version is deployed and verified in "green". Then the traffic is switched entirely. Rollback means switching back. The downside: double the resources, and a shared database requires compatible migrations;
• Dark launch / behind a flag. The code is already in production but the function is off or visible only to the team.

Rollback or fix-forward? Rollback means returning the previous version: fast and predictable when the impact is serious and the cause is not obvious. Fix-forward means shipping a fix quickly: appropriate when the cause is known, the fix is small and a rollback is risky (for example, a data migration already applied is incompatible with the old code). That is why migrations are made compatible: first add the new, then switch over, and only later remove the old (expand / contract).

The condition for all of this is criteria before the start: which metrics and which thresholds count as a signal to stop the rollout. These decisions are made before the release, not under the pressure of an incident.

After the release: monitoring, feedback, blameless post-mortem

A release does not end with the deploy. Some defects show up only on real data, devices and load, so QA looks at production as one more environment to observe. This approach is called shift-right.

What to monitor right after a release.

• Technical signals: error rate (5xx), latency (p95/p99), app crashes, errors in client logs;
• Business KPIs: conversion, orders per hour, successful payments, sign-ups. A release may produce no errors and still break value (a button disappeared, a funnel step does not fire). Compare with the same time of the previous week to rule out seasonality;
• User reports: support tickets, app store reviews, social media mentions. A sharp growth of similar complaints after a release is a strong signal. They are grouped by theme, reproduced, and an automated test is added so the bug does not return;
• Alerts and dashboards: useful only when thresholds make sense; a noisy alert gets ignored.

Smoke in production. A short set of safe checks after a deploy (login, home page, product view) with a dedicated test account that does not spoil real data or distort metrics.

Blameless post-mortem. After an incident the team gathers not to find someone to blame but to understand why the system and the process allowed it. Structure: a timeline of events, the user impact, the root cause (often several), what worked well, and action items with owners and deadlines (a test, an alert, a release checklist item). People share details honestly only when they do not fear punishment. For QA it is a chance to ask: why did the tests not catch this and what should be added.