Backend basics: LB, cache, DB, queues

Client-server and the load balancer

The basic diagram you must be able to draw and explain:

Client -> DNS -> Load Balancer -> [App 1, App 2, App 3] -> Cache
                                         |                 |
                                         +------> DB <-----+
                                         +------> Queue -> Workers

A load balancer distributes requests between several instances of a service. Algorithms: round robin, least connections, a hash of the IP or a key. An LB also does health checks and stops sending traffic to an unhealthy instance.

A stateless service doesn't keep the client's state in memory. The session lives in a token or in a shared store. Then any instance can handle any request, and the service scales easily.

Sticky sessions bind a client to one instance. This is simpler for stateful logic, but when the instance fails the session is lost, and the load is distributed worse.

What QA tests here:

• A scenario where one instance fails in the middle of a flow: does the user continue working.
• Consecutive requests land on different instances: is there no in-memory state that gets "lost". A typical bug: the first request created a payment draft in the memory of instance 1, and the second landed on instance 2 and got 404.
• A rolling deploy: during an update some instances have the new version and some the old. Are they compatible with each other.
• LB timeouts: a long request (a report, an export) may be cut off at the LB before the service responds.

Cache: where it stands and what breaks in it

A cache keeps frequently read data in fast storage (for example Redis), so as not to go to the DB every time.

The main patterns:

• Cache-aside: the service looks in the cache, and if it isn't there, reads from the DB and puts it in the cache. The most common.
• Write-through: a write goes to the cache and the DB at once.
• TTL: an entry in the cache lives for a limited time and then disappears.

The hardest part is invalidation. When data changed in the DB, the cache may still return the old value. For a product catalogue, 30 seconds of a stale price is often acceptable. For an account balance, it isn't.

What can break and what to test:

1. Stale data. After a transfer the balance in the UI is old, because it is read from the cache. The test: change the data and check that a read returns the new value within the promised time.
2. A wrong cache key. The key doesn't take the user or currency into account, and one user sees another's data. That is already a security incident.
3. Cache stampede. A popular key expired, and thousands of requests go to the DB at once. Checked with a load test.
4. The cache is unavailable. The service must work more slowly, but work, not fail.

For tests a cache creates one more problem: a test reads data that another test or a previous run put into the cache. So in test environments it is useful to have a way to reset the cache, or short TTLs.

The database: indexes, replication, sharding

An index is a structure that lets you quickly find rows without a full table scan. An index speeds up reads, but slows down writes and takes space. For QA this matters in two places: slow queries on large volumes of data (on staging with 1000 rows everything is fast, in production with 50 million it isn't) and unique indexes as protection against duplicates.

CREATE UNIQUE INDEX ux_transfers_idem
  ON transfers (client_id, idempotency_key);

Such an index guarantees at the DB level that the same transfer request won't be written twice.

Replication means copies of the DB. The primary accepts writes, and replicas serve reads. Replication is usually asynchronous, so there is replication lag: just-written data may not be on the replica yet. A typical bug: a user created a transfer, the history page reads from a replica, and the transfer isn't there yet. What to test: reading right after a write, and behaviour under an artificial replication delay.

Sharding is distributing data between several DBs by a key, for example by user_id. Each shard stores part of the data. The difficulties: queries across several shards, transactions between shards (a transfer between users on different shards), rebalancing. For testing it is important to have data on different shards, not only on one.

Transactions and isolation. A transfer is a debit and a credit that must happen together. Concurrent transfers from one account may produce a negative balance if there is no locking or the right isolation level. This is checked with parallel requests in a test.

Message queues and delivery guarantees

A queue (Kafka, RabbitMQ, SQS) decouples services: a producer puts a message in, and a consumer processes it later. This gives resilience to peaks and independence of services, but makes the system asynchronous.

Delivery guarantees:

• At-most-once: a message may be lost, but won't be duplicated.
• At-least-once: a message won't be lost, but may arrive twice. The most common variant.
• Exactly-once: hard to achieve across a whole system. In practice you do at-least-once plus an idempotent consumer.

From at-least-once follows the main rule: the consumer must be idempotent. If the message "credit 30 to the account" arrives twice, the money must be credited once. Usually a consumer stores the ids of processed messages and skips duplicates.

Other concepts:

• DLQ (dead letter queue): where messages go that couldn't be processed after N attempts. They can't just be ignored: in fintech this is money stuck in limbo.
• Order. In Kafka order is guaranteed only within a partition. If events of one account go to different partitions, a "debit" may be processed before a "top-up".
• Consumer lag: how many messages are waiting to be processed. A metric for monitoring and for tests under load.

What to test: a duplicate message, a message out of order, a consumer failing in the middle of processing, a poison message that must go to the DLQ and not block the queue.