Documentation and contracts: OpenAPI, JSON Schema, contract testing

Swagger/OpenAPI in testing

OpenAPI is a specification for describing REST APIs, and Swagger is a set of tools around it, including the interactive Swagger UI. The specification describes endpoints, methods, parameters, request and response schemas, response codes and authorisation. For QA it is the main source of requirements for the API.

Through Swagger UI you can quickly execute a request right from the browser, authorise and see the real response. The specification can be imported into Postman to automatically get a collection of all the requests. I also take schemas from it for validating responses in automated tests.

During testing I check whether the API's real behaviour matches the documentation: field types, whether required, error codes. Discrepancies are either a bug in the code or a bug in the documentation, and both are worth filing. Contract testing is built on the specification, and API clients are generated from it.

OpenAPI as a source of tests

An OpenAPI specification describes endpoints, parameters, body schemas, response codes and authorisation. For me it is not only documentation but raw material for automated tests.

How I use it in practice:

• I take schemas from components/schemas and validate responses in automated tests with them, instead of rewriting the structure by hand.
• I generate TypeScript types from the specification (for example, openapi-typescript), and a contract change breaks the compilation of tests.
• I run a specification linter in CI to catch incomplete descriptions: missing error codes, fields without a type.
• I compare the specification between versions to see changes that break clients.

Typical discrepancies I find: a field marked as required but null arrives; 400 is documented, but in reality it is 422; there is no description of errors in the specification at all; an enum in the documentation doesn't match the real values.

Transfer:
  type: object
  required: [id, status, amount]
  properties:
    id: { type: string }
    status: { type: string, enum: [pending, completed, failed] }
    amount: { type: number }

Schema validation in code: ajv and zod

In Postman a schema is checked by jsonSchema, and in TypeScript automated tests ajv (a JSON Schema validator) or zod (schemas right in TypeScript with type inference) are usually used.

import { test, expect } from '@playwright/test';
import { z } from 'zod';

const Transfer = z.object({
  id: z.string(),
  status: z.enum(['pending', 'completed', 'failed']),
  amount: z.number().positive(),
  currency: z.string().regex(/^[A-Z]{3}$/),
}).strict();

test('create transfer matches contract', async ({ request }) => {
  const res = await request.post('/v1/transfers', {
    data: { toAccount: 'UA21', amount: 150, currency: 'UAH' },
  });
  expect(res.status()).toBe(201);
  const parsed = Transfer.safeParse(await res.json());
  expect(parsed.success, JSON.stringify(parsed.error?.issues)).toBe(true);
});

.strict() in zod works like additionalProperties: false: an extra field fails the test. The choice of tool is simple: if schemas already exist in OpenAPI or JSON Schema, I take ajv and don't duplicate them; if there are no schemas, zod is more convenient, because it gives both the check and the type at once.

Consumer-driven contracts and breaking changes

Contract testing checks that two services understand each other, without bringing up the whole system. In the consumer-driven contracts approach (the Pact tool) the consumer, for example a front end or another microservice, describes in a test which requests it makes and which fields it expects. A contract is generated from this, it is published to the Pact Broker, and the provider in its own CI verifies that the real API fulfils this contract.

Advantages: fast, isolated, and the provider learns that it broke a specific consumer before deployment. Pact checks only what the consumer really uses, so it isn't a replacement for the provider's functional tests.

Breaking vs non-breaking changes:

A nuance: a new value in an enum is formally an "addition", but may break a client that handles only known values.