Scaling and reliability

Vertical and horizontal scaling

Vertical scaling is a bigger machine: more CPU, more memory. Simple, no code changes, but there is a ceiling and a single point of failure.

Horizontal scaling is more instances behind a load balancer. Almost no ceiling and more resilient to failures, but it requires stateless services, shared storage for state and complicates the system.

Autoscaling adds instances by a metric: CPU, the number of requests, queue length. Important nuances: a new instance doesn't start instantly, and the DB often scales worse than services.

What QA tests in the context of scaling:

A load test: behaviour under the expected load: p95 and p99 latency, error rate.
A stress test: where and how the system breaks: whether it degrades gradually or fails completely.
A spike test: a sudden peak, for example a sale or a salary payout at 9:00. Does autoscaling keep up.
A soak test: several hours of load to find memory and connection leaks.

Tools: k6, Gatling, JMeter, Locust. The choice of tool is less important than a realistic load profile and an environment close to production.

A typical mistake: running load on staging with an empty database. Queries that are fast on 1000 rows become slow on 50 million. A realistic data volume is needed.

In an interview: Horizontal scaling requires stateless services and gives resilience, while vertical is simpler but has a ceiling. As QA I check scaling with load, stress, spike and soak tests with a realistic data volume and look at p95, p99 and the error rate.

Idempotency, retries and rate limiting

Idempotency means a repeated call of an operation gives the same result as the first. GET, PUT, DELETE are idempotent by the HTTP specification, and POST isn't. For payments POST is made idempotent too through an idempotency key: the client generates a unique key and passes it in a header. The server remembers the result by the key and on a repeat returns the same result without creating a new payment.

Why: the network is unreliable. The client sent a transfer, the response got lost, the client repeated. Without a key, a double debit.

Test cases for an idempotency key:

A repeat with the same key and the same body: the same transfer, one debit.
The same key with a different body: an error, not silent ignoring.
Two parallel requests with one key: one transfer.
A repeat after the key's retention period has expired: behaviour per the specification.

Retries with exponential backoff and jitter. The client repeats a request with a growing pause (1, 2, 4 seconds) and a random offset, so that thousands of clients don't repeat at the same time. Only idempotent operations may be repeated.

Rate limiting restricts the number of requests from a client over time, for example 100 requests per minute. Exceeding it gets a 429 Too Many Requests response, often with a Retry-After header. Algorithms: token bucket, sliding window. What to test: the limit boundary, the window reset, limits for different clients are independent, the client reacts to 429 correctly.

In an interview: For payments I always test an idempotency key: a repeat, parallel requests with one key and the same key with a different body. Separately I check rate limiting at the limit boundary and that the client handles 429 correctly with backoff, rather than repeating the request instantly.

CAP and reliability at a basic level

The CAP theorem says: during a network partition (P) a distributed system must choose between consistency (C) and availability (A). Partitions do happen in a real network, so the choice is in fact between C and A at the moment of a failure.

CP: better to refuse an operation than to return inconsistent data. Example: debiting an account. A failure is better than a double debit.
AP: better to respond even if the data is slightly stale. Example: a product catalogue, the number of likes, recommendations.

The practical conclusion for QA: you need to know which behaviour is chosen for each function and test exactly that. If the balance is CP, then when the ledger is unavailable the API must return an error, not an old balance from the cache.

Reliability patterns worth knowing:

A timeout on every call to another service. Without a timeout one slow service blocks all threads.
A circuit breaker: after a series of errors it stops calling a dependency for a while and immediately returns a fallback or an error.
Graceful degradation: if the recommendations service is unavailable, the payment page still works.
A bulkhead: isolation of resources, so that the failure of one function doesn't take all the connections.

How to test this: chaos testing and fault injection. Delay a dependency's responses, kill an instance, break the DB connection, and check that the system behaves as intended. It is worth starting on staging and with controlled experiments.

In an interview: In practice CAP means a choice between consistency and availability during a network failure: for a balance I expect CP, for a catalogue AP is acceptable. I test this behaviour through fault injection: delays, failing dependencies, and I check timeouts, the circuit breaker and degradation of non-critical functions.