Runner, data-driven tests and Newman

Collection Runner and data-driven tests

The Collection Runner runs a whole collection or folder sequentially, with the chosen environment, number of iterations and a delay between requests. The result is a pass/fail table for each pm.test.

Data-driven. In the Runner you attach a CSV or JSON file: one row is one iteration, and the columns become variables. Convenient for boundary values and negative cases:

amount,currency,expectedStatus
150,UAH,201
0,UAH,400
-10,UAH,400
1000000000,UAH,422
150,XXX,400

In the request body: {{amount}} and {{currency}}, and in the test:

pm.test(`amount=${pm.iterationData.get('amount')}`, () => {
  pm.response.to.have.status(Number(pm.iterationData.get('expectedStatus')));
});

The test name contains the value from the iteration, so the report shows at which row it failed. Number(...) is needed because values from CSV arrive as strings. That way one request covers five cases, and new boundaries are added as a row in the file without changing scripts.

In an interview: The Collection Runner runs a collection with an environment and iterations. For data-driven runs I attach a CSV or JSON where each row is an iteration with input data and the expected status, and that is how I cover boundary and negative values.

Newman and CI

Newman is a CLI runner of collections without a UI, for CI. The collection and environment are exported to JSON and put in the repository:

npm i -g newman newman-reporter-htmlextra
newman run bank-api.postman_collection.json \
  -e stage.postman_environment.json \
  -d transfers.csv \
  --env-var "password=$API_PASSWORD" \
  -r cli,htmlextra,junit

If at least one test fails, Newman exits with a non-zero code, and the CI job turns red. The JUnit report is picked up by Jenkins, GitLab and GitHub Actions. Secrets are passed through --env-var from the CI secrets, not stored in the environment file.

An alternative from Postman itself is the Postman CLI (postman collection run), which works with collections from the cloud workspace.

The HTML report from htmlextra is convenient to save as a job artifact: when something fails at night, the raw requests and responses are visible without rerunning.

In an interview: On CI I run the collection through newman run with an exported environment and a JUnit reporter, and pass secrets through --env-var. A failed test gives a non-zero exit code, and the pipeline turns red.

Postman's limits

A large set of API tests with logic, data factories and git review is easier to maintain in code: Playwright request, Cypress cy.request, RestAssured. Postman is strong for exploring an API, quick checks and shared documentation with developers.

The reasons regression is moved into code:

A JSON collection reads poorly in a diff, and reviewing script changes is inconvenient.
It is hard to reuse logic: helpers, test data factories, shared fixtures.
In code, API tests live next to UI tests, with the same CI, reporting and types.
In API tests in TypeScript you can type responses and catch contract changes at compile time.

At the same time I don't reject Postman: when a new endpoint arrives, I first explore it in Postman, and then move stable scenarios into automated tests. Importing OpenAPI into Postman also saves time at the start.

In an interview: Postman is for exploration and quick checks, and a large regression suite I keep in code, for example on Playwright request, next to the UI tests and with git review. That makes it easier to reuse logic and maintain the tests.