CI, parallelization, reports

running in the pipeline

How do you run Cypress in GitHub Actions?

The simplest is the official action cypress-io/github-action: it installs dependencies with a cache, starts the application if needed, waits for it and runs the tests. Artifacts with screenshots are uploaded on failure.

name: e2e
on: [pull_request]

jobs:
  cypress:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        with:
          browser: chrome
          start: npm start
          wait-on: 'http://localhost:3000'
        env:
          CYPRESS_USER_PASSWORD: ${{ secrets.USER_PASSWORD }}
      - uses: actions/upload-artifact@v4
        if: failure()
        with:
          name: screenshots
          path: cypress/screenshots

In an interview: I use the official action cypress-io github-action: it installs dependencies with a cache, brings up the application, waits for it and runs the tests. Secrets are passed from the repository's secrets, and on failure screenshots are saved as artifacts. Tests run on every pull request.

And in Playwright:

- run: npm ci
- run: npx playwright install --with-deps
- run: npx playwright test
- uses: actions/upload-artifact@v4
  if: always()
  with:
    name: playwright-report
    path: playwright-report

There is also an official Docker image with browsers: mcr.microsoft.com/playwright.

How do you parallelise Cypress tests?

On one machine Cypress doesn't run in parallel: one process, one browser, specs one after another. Parallelisation is only across several machines.

Cypress Cloud: cypress run --record --parallel distributes specs between machines taking duration into account. This is a paid service after a free limit.
For free: a matrix in CI plus the cypress-split plugin, which divides specs between runs.

The unit of distribution is a spec file, so large specs are better split up.

strategy:
  matrix:
    containers: [1, 2, 3, 4]
steps:
  - uses: cypress-io/github-action@v6
    env:
      SPLIT: 4
      SPLIT_INDEX: ${{ strategy.job-index }}

In an interview: On one machine Cypress runs specs sequentially, so parallelisation happens only across several machines. The official way is Cypress Cloud with the record and parallel flags, but it is paid. For free you can build a matrix in CI and split the specs with the cypress-split plugin. The unit of distribution is a spec file, so I keep specs small.

And in Playwright:

Parallelisation is free and built in: workers on one machine (half of the CPU cores by default) plus --shard=1/4 between machines and merge-reports for merging reports. The unit of distribution is a test, not a file.

What reports, screenshots and videos does Cypress give you?

In cypress run a screenshot on failure is taken automatically. Video has been off by default since Cypress 13 and is turned on with video: true.

Reports: the built-in spec reporter in the console, mochawesome for HTML (with a separate report merge after the run), allure-cypress for Allure with history and trends, JUnit XML for CI.

// cypress.config.ts
export default defineConfig({
  video: true,
  screenshotOnRunFailure: true,
  reporter: 'mochawesome',
  reporterOptions: {
    reportDir: 'cypress/reports',
    overwrite: false,
    html: false,
    json: true,
  },
});

In an interview: On a failure in run mode a screenshot is taken automatically, and video has to be turned on explicitly, because it has been off since version thirteen. For HTML reports I use mochawesome, and if I need history and trends, Allure through allure-cypress. For CI systems there is JUnit XML.

And in Playwright:

An HTML report with a built-in Trace Viewer comes out of the box. Screenshots, video and traces are configured in use: screenshot, video, trace. Allure goes through allure-playwright, JUnit through a built-in reporter.

Flaky tests in Cypress and retries

The most frequent causes: fixed cy.wait instead of waiting for conditions, races with network requests, animations, tests depending on each other and on shared data. Also unstable selectors, an element re-rendering after it was found, and a slow CI environment.

The treatment: wait for requests through cy.intercept and aliases, build chains with assertions that are retried, isolate tests and create data through the API. Each test must start from a known state. For diagnostics: video, screenshots and Test Replay in Cypress Cloud.

Retries are configured in the config: retries: { runMode: 2, openMode: 0 }, where runMode is for cypress run on CI and openMode is for interactive mode. You can set retries for an individual test or a describe block too. But retries only mask the problem, so flaky tests must be tracked and the root cause fixed.