GitHub Actions for a test framework

A workflow skeleton for Playwright in GitHub Actions

A workflow is a YAML file in .github/workflows/. It consists of jobs that by default run in parallel, each on its own runner. A job is a sequence of steps on one machine (a shared file system). A step is either run: (a shell command) or uses: (a ready-made action).

Triggers (on:): pull_request (a PR), push (with a branch filter), schedule (cron, always in UTC), workflow_dispatch (a manual run).

name: e2e
on:
  pull_request:
  push:
    branches: [main]
  schedule:
    - cron: '0 2 * * *'
  workflow_dispatch:

jobs:
  e2e:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: npm
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      - run: npx playwright test
      - uses: actions/upload-artifact@v4
        if: ${{ !cancelled() }}
        with:
          name: playwright-report
          path: playwright-report/
          retention-days: 14

What matters here: timeout-minutes protects against hangs (the default is 360 minutes); npm ci installs dependencies strictly from the lock file; if: ${{ !cancelled() }} uploads the report even after the tests failed, because by default a step has the condition success() and is skipped after a red step.

Cache, matrix, concurrency and sharding

Dependency cache. actions/setup-node@v4 with cache: npm caches the global npm cache (~/.npm) under a key derived from package-lock.json; npm ci still runs, but takes packages from disk. Caching node_modules itself is usually discouraged. For your own caches use actions/cache@v4 with a key that changes with the lock file: key: pw-${{ runner.os }}-${{ hashFiles('**/package-lock.json') }}. Playwright browsers live in ~/.cache/ms-playwright; after a cache hit the system dependencies still must be installed (npx playwright install-deps) because they are not cached.

Matrix. One job multiplied by parameters: browsers, Node versions, shard numbers. Each combination is a separate job on a separate runner. By default fail-fast: true: when one combination fails, the others still running are cancelled. For the full picture set fail-fast: false.

jobs:
  e2e:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2, 3, 4]
    steps:
      # ... checkout, setup-node, npm ci, install browsers
      - run: npx playwright test --shard=${{ matrix.shard }}/4 --reporter=blob
      - uses: actions/upload-artifact@v4
        if: ${{ !cancelled() }}
        with:
          name: blob-report-${{ matrix.shard }}
          path: blob-report
          retention-days: 1

In upload-artifact@v4 artifact names within a run must be unique, hence the shard number in the name.

Concurrency. So that a new push to a PR cancels the outdated run:

concurrency:
  group: e2e-${{ github.ref }}
  cancel-in-progress: true

For main cancelling is usually not enabled (cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}) so as not to lose the result of a run on merged commits.

Secrets, service containers, GitLab CI and Jenkins

Secrets. They are stored in Settings → Secrets and variables, read in a workflow as ${{ secrets.API_TOKEN }} and masked in logs. Pass them through env: only to the step that needs them. For PRs from forks secrets are not passed (protection against theft); the dangerous pull_request_target with a checkout of foreign code should be avoided. Non-sensitive settings (environment URL, flags) are vars, not secrets.

Service containers. For integration tests that need a database, add services: to the job: the container starts before the steps and stops after the job.

jobs:
  api-tests:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:16
        env:
          POSTGRES_PASSWORD: test
          POSTGRES_DB: app_test
        ports: ['5432:5432']
        options: >-
          --health-cmd "pg_isready -U postgres"
          --health-interval 5s
          --health-timeout 5s
          --health-retries 10
    env:
      DATABASE_URL: postgres://postgres:test@localhost:5432/app_test
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: npm
      - run: npm ci
      - run: npm test

The --health-* options make the job wait until the DB is really ready. If the job itself runs in a container (container:), the service is reached by its name (postgres); on a runner, via localhost and the port mapping.

Other systems: the same ideas. GitLab CI: .gitlab-ci.yml, stages, parallel: 4 instead of a matrix, cache:, artifacts: when: always, services:. Jenkins: a Jenkinsfile with stages and parallel, post { always { ... } } for reports, agents instead of runners. The syntax differs, the approach does not: triggers, cache, parallelism, artifacts, secrets.