Playwright

Playwright provide us the ability to do end-to-end testing, headless browser testing, amongst others; local visual regression testing as an alternative to Chromatic's cloud-based approach.

Use Playwright when you need to:

• Faster feedback loops during active development
• Full control over test execution and baselines, e.g. snapshots or screenshots
• OS specific visual testing

Requirements

• Docker

Visual regression testing locally

Visual regression tests are operating system dependent.

Screenshots generated on macOS will differ from Linux or Windows due to font rendering, anti-aliasing, and browser engine differences.

To mitigate differences across operating systems, our visual regression tests are Dockerized for consistent visual regression testing.

Docker (Ubuntu Linux)

All visual tests run in Docker (Ubuntu Linux) by default for consistency with CI:

yarn test:visual          # Run tests
yarn test:visual:update   # Update snapshots
yarn test:visual:ui       # UI mode (opens at http://localhost:8282)
yarn test:visual:report   # View HTML report
yarn test:visual names    # List the component names you can scope a run to

Snapshots are saved with -linux suffix to match CI environments.

For this reason at time of writing, the following snaptshots are ignored:

# The snapshots which are generally persisted for comparison
**.ts-snapshots

# These are generated during test runner, generally ignored
tests/**/*-actual.png
tests/**/*-diff.png
test-results

Generating snapshots for comparison

You MUST generate the snapshots you want to compare against. At time of writing, the team's using Chromatic for visual regression testing.

To generate snapshots, you'll have to manually checkout/switch to the target commit in history and run the test:visual to generate it and return back to your ongoing feature branch. Alternatively, you can store snapshots separately and place them at your need.

Hypothetically, you could use the following workflow to facilitate:

# Switch to a target/stable version of your liking
git checkout <COMMIT-STABLE-VERSION>
# Generate the snapshots against your target version
yarn test:visual
# Check if snapshots generated?
find ./tests -type f -name '**.ts-snapshots*'
# Once happy return to feature branch
git checkout <FEATURE-BRANCH>
# Run the tests
yarn test:visual

When intentional visual changes are made, update baselines:

yarn test:visual:update

Running tests locally

Execute all visual regression tests by running:

yarn test:visual

Alternatively, launch UI mode for a browser like experience with timeline, console logs, etc.

yarn test:visual:ui

Run a single component

The full suite takes around 20 minutes, so scope it to what you are working on. Pass a component name to either command and only the specs guarding that component run:

yarn test:visual CodeBlock          # run CodeBlock's visual tests
yarn test:visual:update CodeBlock   # update only CodeBlock's baselines

Names are case- and separator-insensitive, so CodeBlock, codeblock and code-block are equivalent. You can also name:

What you type	What runs
a component (IconButton)	every spec that declares @covers src/components/IconButton — including other components' specs that render it
a spec file name (inputfield)	that spec
a suite directory (buttons)	every spec in tests/buttons/
a 3+ character fragment (card)	every name containing it, listed back to you before the run

Resolution is driven by the @covers directives at the top of each spec, the same source of truth CI uses to scope its runs. To see every name you can use:

yarn test:visual names

An unknown name exits immediately rather than falling back to the full suite, so a typo costs a second instead of 20 minutes.

Run specific test files

Paths and Playwright flags still work, and can be mixed with names:

yarn test:visual tests/buttons/overview.spec.ts
yarn test:visual CodeBlock -g "hovered controls"

Show report

To review test reports as HTML run:

yarn test:visual:report

Configuration

The Playwright configuration file contains default configuration values, which you can extend, modify or personalize.

For example, you can modify the default number of workers:

  ...

  workers: process.env.CI ? 4 : undefined,

Migration to Chromatic

Playwright tests can be easily migrated to Chromatic as currently (per the example provided), both leverage Storybook stories. Meaning that you can start with Playwright for local development and transition to Chromatic as collaboration or any requirements demand.

Here's a quick example of migration switch:

import { test, expect } from "@playwright/test"; 
import { test, expect } from "@chromatic-com/playwright"; 

Learn more

• Test snapshots
• Best practices