Playwright Retries
Playwright Retries
A retry runs a failed test again, to see if it passes the second time.
Retries help you tell a real bug from a test that only fails sometimes.
Retries and Flaky Tests
A flaky test is a test that sometimes passes and sometimes fails, without any change to the code.
A test that fails first and then passes on a retry is reported as flaky.
Set the number of retries in the config:
Example
playwright.config.js
// @ts-check
import { defineConfig, devices } from '@playwright/test';
export default defineConfig({
testDir: './tests',
fullyParallel: true,
retries: 1,
reporter: 'html',
use: {
baseURL: 'https://www.w3schools.com/playwright/practice/',
trace: 'on-first-retry',
},
projects: [
{ name: 'chromium', use: { ...devices['Desktop Chrome'] } },
],
});
This test is flaky on purpose.
testInfo is the second argument of the test function.
It holds information about the running test.
testInfo.retry is 0 on the first run and 1 on the first retry, so the test fails once and then passes:
Example
tests/flaky.spec.js
// @ts-check
import { test, expect } from '@playwright/test';
test('has title', async ({ page }) => {
await page.goto('todo.html');
await expect(page).toHaveTitle(/Todo List/);
});
test('fails on the first try', async ({ page }, testInfo) => {
await page.goto('todo.html');
console.log('Attempt', testInfo.retry + 1);
expect(testInfo.retry).toBeGreaterThan(0);
});
Output:
Running 2 tests using 2 workers
[1/2] [chromium] › tests/flaky.spec.js:4:1 › has title
[2/2] [chromium] › tests/flaky.spec.js:9:1 › fails on the first try
[chromium] › tests/flaky.spec.js:9:1 › fails on the first try
Attempt 1
[3/2] (retries) [chromium] › tests/flaky.spec.js:9:1 › fails on the first try (retry #1)
Attempt 2
1) [chromium] › tests/flaky.spec.js:9:1 › fails on the first try ─────────────────────────────────
Error: expect(received).toBeGreaterThan(expected)
Expected: > 0
Received: 0
1 flaky
[chromium] › tests/flaky.spec.js:9:1 › fails on the first try ──────────────────────────────────
1 passed (3.0s)
The retry is shown as [3/2] (retries), with (retry #1) after the test name.
The run passes, but the summary says 1 flaky, and the HTML report marks the test with a warning sign:
The retry also recorded a trace, because the config has trace: 'on-first-retry'.
Click View Trace to see what happened.
Set retries for one run on the command line:
Example
npx playwright test --retries=2
Note: Retries hide flaky tests, but they do not fix them.
Look at every flaky test, and fix the cause.
A common cause is a test that reads the page before it is ready, because it has no assertion that waits.
Use --fail-on-flaky-tests to make flaky tests fail the run.
Repeat Tests
To find a flaky test, run it many times with --repeat-each:
Example
npx playwright test --repeat-each=3 -g "has title"
Output:
Running 3 tests using 3 workers
[1/3] [chromium] › tests/flaky.spec.js:4:1 › has title
[2/3] [chromium] › tests/flaky.spec.js:4:1 › has title
[3/3] [chromium] › tests/flaky.spec.js:4:1 › has title
3 passed (2.1s)
-g "has title" runs only the tests that have "has title" in their title.
A test that passes 100 times in a row with --repeat-each=100 is probably not flaky.