Performance and reliability

Kinds of load tests and percentiles

Type	What we do	What we look for
Load	The expected working load for a long time	Whether we hold the SLA under normal conditions
Stress	Gradually raise the load above normal	The breaking point and how the system degrades
Spike	A sudden jump, for example 10 times in a minute	Whether it withstands the spike and recovers
Soak	A moderate load for many hours	Memory leaks, connection pool exhaustion

Tools: k6, JMeter, Gatling, Locust.

Why not the average. An average time of 200 ms may hide that every twentieth request takes 5 seconds. So percentiles are used: p95 means 95% of requests are faster than this value, p99 means 99%. An SLA is usually worded like this: "p95 < 500 ms at 200 RPS, errors under 0.1%". The p99 tail often hurts the most, because one page makes dozens of requests, and someone almost always hits a slow one.

It is important to test on an environment close to production with realistic data: on an empty database everything is fast.

In an interview: Load checks the expected load, stress looks for the limit, spike a sudden burst, and soak stability over a long time. I evaluate p95 and p99, not the average, because the average hides the slow tail.

Timeouts, retries and idempotency keys

Timeouts. Every call to another service must have a timeout, otherwise one hung service holds threads and connections, and everything falls down. I test that with a slow dependency the client gets a controlled error (often a 504 from the gateway) in a predictable time and doesn't hang for a minute.

Retries with backoff. Only temporary errors may be repeated: a timeout, 429, 502, 503, 504. On 400 or 422 a retry is pointless: the result won't change. Retries are done with exponential backoff (1 s, 2 s, 4 s) and jitter, a random offset, so that thousands of clients don't hit the server at once. The number of attempts is limited.

Idempotency keys. Retrying a non-idempotent POST is dangerous: a timeout doesn't mean the request wasn't executed, maybe only the response was lost. So the client generates an Idempotency-Key once per operation and passes it in all repeats, and the server returns the stored result instead of a second payment.

A fintech test case: emulate a response timeout (for example through a proxy or a mock of the dependency), wait for the client's retry and check in the database that there is one transfer.

In an interview: I check that calls have timeouts, and that retries happen only on temporary errors with exponential backoff and jitter. For payments every retry goes with the same Idempotency-Key, and I check that there is one transaction in the database.

429, 503, Retry-After and the circuit breaker

429 Too Many Requests means the client exceeded a limit. 503 Service Unavailable means the service temporarily can't process the request: overload, a deploy, a dependency is down. Both may have the Retry-After header: the number of seconds or a date after which it is worth repeating:

HTTP/1.1 429 Too Many Requests
Retry-After: 30

A correct client respects Retry-After and doesn't retry earlier. What I check: the limit fires at the right threshold; the header is present and has a valid value; after the pause requests go through again; the client (our front end or service) shows a clear message instead of spinning a loader forever.

A circuit breaker is a fuse between services. If a dependency fails again and again, the breaker "opens" (the open state), and calls to it immediately return an error or a fallback without waiting for the timeout. After some time it lets a few trial requests through (half-open): if they succeed, it closes again (closed). This stops the failure of one service from bringing down the whole system in a cascade.

For QA these are resilience scenarios: turn off or slow down a dependency and check that the main service responds quickly with a controlled error or a fallback, and that after the dependency recovers everything returns to normal without manual intervention.

In an interview: On 429 and 503 I check for Retry-After and that the client respects it. A circuit breaker, after a series of failures, stops going to the dependency and quickly returns an error or a fallback, and I test this by turning off the dependency and checking recovery.