Visual Regression Testing with Playwright and GitHub Actions

Even if the website is built in a perfect way, has the nicest content and is SEO optimised, it’ll always require maintenance. Programming languages, libraries, frameworks, operating systems evolve constantly and receive security updates. Older versions become unsupported. The only way to keep a website up and running is to keep all its parts up to date.

At Amazee Labs we take maintenance seriously. All our websites receive updates regularly. In order to make the testing of the upgrades easier, we create as many automated tests as possible during the development phase. We use all kinds of tests: static, unit, visual regression, integration (end-to-end). Whatever is suitable for each individual case.

Visual Regression Testing

Recently, we mostly build decoupled websites with modern React frontends. When it comes to visual testing, our best friend is Chromatic. It can run visual tests for individual components or full pages, for anything we implement in Storybook. It works perfectly for decoupled frontends. But what if we want to visually test a Drupal-only project, or the Drupal admin UI? Chromatic can’t help with that.

Our favourite tool for end-to-end testing is Playwright. It has a built-in feature for making and comparing the screenshots:

It can even report the visual differences in nice ways.

That’s it! We thought 😅

Then came the fun part.

What can go wrong

Some browsers can render websites slightly differently depending on the operating system. More than that, the screenshots can have tiny distinctions on the same OS but different CPU architectures. Mac Intel and Mac ARM will produce different snapshots. This means we can’t reliably run the tests. Screenshots made locally won’t be the same with the ones made in CI or on a colleague’s machine.

One way to solve this would be to run tests in Docker with --platform linux/amd64flag. But that would make the whole testing process heavier and slower.

Another option would be to execute tests in CI and only in CI. We just need to make it easy to get the test reports and update the screenshots.

The setup

We execute most of our automated tests in GitHub Actions, which has a variety of ready-to-use actions. The one we need is the upload-artifact.

Playwright needs to be configured to produce HTML reports:

And here’s the GitHub workflow steps:

So, in the case of failing tests, we download the report, check the diffs, and then either

- fix the project, so that the screenshots match again

- or download the updated screenshots and commit them to the project.

That’s it! Now for sure 🎉

In conclusion, visual regression testing with Playwright and GitHub Actions offers a robust solution for maintaining website integrity. By embracing automated testing, as we do, your website can receive constant updates, stay secure, and remain stable during upgrades. Take the next step towards maintaining a reliable website by implementing these strategies in your development workflow today!