Playwright Continuous Integration
Playwright in Continuous Integration
Continuous Integration (CI) means a server runs your tests automatically, every time someone pushes code or opens a pull request.
To push code means to send your commits to the shared Git repository, and a pull request asks to merge a change into the main branch.
A broken change is caught before it is merged, not after it reaches users.
Playwright runs in any CI system: GitHub Actions, GitLab CI, Azure Pipelines, Jenkins, CircleCI and others.
This chapter uses GitHub Actions, because npm init playwright can set it up for you.
The GitHub Actions Workflow
When you create a project with npm init playwright@latest, it asks if you want a GitHub Actions workflow.
If you answer yes, it creates this file:
Example
.github/workflows/playwright.yml
name: Playwright Tests
on:
push:
branches: [ main, master ]
pull_request:
branches: [ main, master ]
jobs:
test:
timeout-minutes: 60
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- uses: actions/setup-node@v6
with:
node-version: lts/*
- name: Install dependencies
run: npm ci
- name: Install Playwright Browsers
run: npx playwright install --with-deps
- name: Run Playwright tests
run: npx playwright test
- uses: actions/upload-artifact@v5
if: ${{ !cancelled() }}
with:
name: playwright-report
path: playwright-report/
retention-days: 30
You can also create the file yourself.
Commit it, push, and GitHub runs it.
Workflow Explained
on: run on every push and pull request to themainormasterbranchruns-on: ubuntu-latest: run on a fresh Linux machine from GitHubactions/checkout: get your codeactions/setup-node: install Node.js (the latest LTS version, LTS means Long Term Support)npm ci: install the exact package versions frompackage-lock.jsonnpx playwright install --with-deps: download the browsers, and the Linux system libraries they neednpx playwright test: run the testsactions/upload-artifact: save the HTML report as an artifact (a file kept after the run), so you can download it from the run pageif: ${{ !cancelled() }}: upload the report even when tests failed, which is exactly when you need it
To read the report, open the workflow run on GitHub, download the playwright-report artifact, unzip it and run npx playwright show-report with the folder name.
Note: Commit package-lock.json.
npm ci needs it, and it makes CI use the same Playwright version as your computer.
CI Settings in the Config
Most CI systems set the environment variable CI.
The config made by npm init playwright uses it to behave differently in CI:
Example
playwright.config.js
// @ts-check
import { defineConfig, devices } from '@playwright/test';
export default defineConfig({
testDir: './tests',
fullyParallel: true,
forbidOnly: !!process.env.CI,
retries: process.env.CI ? 2 : 0,
workers: process.env.CI ? 1 : undefined,
reporter: 'html',
use: {
baseURL: 'https://www.w3schools.com/playwright/practice/',
trace: 'on-first-retry',
},
projects: [
{ name: 'chromium', use: { ...devices['Desktop Chrome'] } },
],
});
| Setting | In CI | Why |
|---|---|---|
forbidOnly | true | Fail the run if someone left a test.only in the code, which would skip all other tests |
retries | 2 | Retry failed tests, and mark tests that pass on retry as flaky (see Retries) |
workers | 1 | Run one test at a time, which is more stable on small CI machines |
trace | on first retry | Record a trace when a test is retried, so you can debug it (see Trace Viewer). This setting is the same everywhere, but only CI has retries |
On your own computer CI is not set, so you get no retries and as many workers as your CPU allows.
If your CI machines are large, you can raise workers, for example to '50%' of the CPU cores.
Sharding
When you have many tests, one machine can be too slow.
Sharding splits the tests into parts, and runs each part on its own machine at the same time.
The --shard option picks one part.
You can try it on your own computer, here with the 5 tests from the Page Object Model chapter:
Example
npx playwright test --shard=1/2
npx playwright test --shard=2/2
Output:
Running 3 tests using 3 workers, shard 1 of 2
[1/3] [chromium] › tests/login.spec.js:12:1 › wrong password shows an error
[2/3] [chromium] › tests/login.spec.js:5:1 › login works
[3/3] [chromium] › tests/todo.spec.js:5:1 › can add todos
3 passed (5.3s)
Running 2 tests using 2 workers, shard 2 of 2
[1/2] [chromium] › tests/todo.spec.js:14:1 › can complete a todo
[2/2] [chromium] › tests/todo.spec.js:25:1 › can delete a todo
2 passed (2.0s)
Together the two shards ran all 5 tests, and no test ran twice.
In GitHub Actions, a matrix is a list of values, and GitHub starts one job for each value.
Here it starts one job per shard:
Example
strategy:
fail-fast: false
matrix:
shardIndex: [1, 2, 3, 4]
shardTotal: [4]
steps:
# ... checkout, setup-node, npm ci, install browsers
- name: Run Playwright tests
run: npx playwright test --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }}
fail-fast: false lets the other shards finish when one shard fails.
Each shard makes its own report.
To get one report for the whole run, use the blob reporter (see File Reporters) in the shards, upload the blob-report folders, and join them in a last job with npx playwright merge-reports.
The Playwright Docker Image
Microsoft publishes a Docker image with Node.js, the browsers and all their system libraries already installed:
mcr.microsoft.com/playwright:v1.65.1-noble
The tag is the Playwright version, and noble is the Ubuntu version (24.04).
Use the same version as @playwright/test in your package.json, or the browsers will not match.
In GitHub Actions, run the job inside the image with container.
Then you can skip the Node.js setup and the browser install:
Example
jobs:
test:
runs-on: ubuntu-latest
container:
image: mcr.microsoft.com/playwright:v1.65.1-noble
steps:
- uses: actions/checkout@v6
- run: npm ci
- run: npx playwright test
The image is also the easiest way to make screenshot baselines on Windows or Mac that match the Linux screenshots in CI.
Tips for CI
- Start with the generated workflow, and change it only when you need to
- Always upload the HTML report, and keep
trace: 'on-first-retry', so you can debug failures that only happen in CI - If CI only needs one browser, install just that one:
npx playwright install --with-deps chromium - Keep secrets, like test passwords, in your CI system's secret store, and read them with
process.env - A flaky test (one that sometimes passes and sometimes fails) is a real problem: fix it instead of adding more retries