Percy AI visual regressions – playwright example

Challenge: automate screenshot comparison for regression testing

Solution: write tests in Playwright and use the smart AI screenshot comparison in percy.ai

Tired of regression bugs on your websites? Would you like to monitor site layout changes periodically and detect any render anomalies in Chrome, Firefox, Safari or Edge? In this article, you will find the answer. We will describe our solution for writing e2e tests in Playwright, generating screenshots in Percy.io and scheduling a daily job using the Jenkins CI tool.

Visual regression testing

During website development or during the introduction of small changes there is always the risk of applying incorrect styling or adding javascript code that will break the layout. We can always focus on manual testing, but this is time consuming and not scalable. A popular solution to regression bugs were screenshots comparison tools, which were comparing pixel by pixel and reporting any differences.

What was the problem with such tools? A lot of false positives – any difference in font rendering or small shifts not visible to the human eye were always reported. Automated workflow required a manual check of every falsely reported test.

Percy – AI to the rescue

Previously time-consuming processes are starting to be automated thanks to the development of computer vision techniques. Smart visual testing is introduced in the percy.io service. Percy provides a way to compare screenshots with the baseline.

Any visual changes are highlighted and can be reviewed, approved, or commented. Percy was acquired in 2020 by BrowserStack, and provides a starter plan with a free tier of 5000 screenshots monthly.

Website screens are compared in a very efficient way, no more long waiting – builds are usually completed in a couple of minutes. Percy uses smart visual techniques and AI processing to exclude false positives and provide only relevant visual changes reports.

Percy can visually test anything that runs in a browser. It’s supports different end-to-end testing frameworks including: Ember, Cypress, TestCafe, Capybara, Puppeteer, Playwright, Protractor, Nightmare, Nightwatch, WebdriverIO and Selenium.

Percy technology

Screenshots are tested in the cloud using the 4 most popular browsers: Chrome, Firefox, Edge and Safari. We can also define width breakpoints to test smaller resolutions. What is nice is that we can also test user scenarios: tests written in Playwright will be executed and screenshots captured at the end of a successful user journey (registration or login process).

Under the hood – Google Kubernetes in Google Cloud Platform is used for visual comparison. New screens are compared to previously approved “Baseline screens”. We can even set an option that all things pushed to master are already approved as a baseline screenshot, which can improve workflow.

The NPM package @percy/cli is used to create snapshots of a website. The already generated DOM html code and assets are send to percy API and site is recreated in the cloud container and rendered again in the most popular browsers. The advantage of this solution is that a website doesn’t need to be publicly available! You can easily just point to your localhost url and a screenshot will be generated successfully!

Tests in playwright

To write end-to-end tests, we decided to choose Playwright, a cross-browser tool implemented by Microsoft. It’s open-source, fast, and minimizes flaky tests by introducing the “auto-wait” feature.

Our test will be very simple – just visit the homepage and trigger the percySnapshot method. After logging in to the Percy dashboard, we will see the comparison of results. Tests are written in JavaScript and use 3 NPM packages: @percy/cli, @percy/playwright and @playwright/test . Here is a package.json :

{
  "devDependencies": {
    "@percy/cli": "^1.3.1",
    "@percy/playwright": "^1.0.4",
    "@playwright/test": "^1.22.2"
  },
  "scripts": {
    "runpercy_test1": "percy exec --config ./percy.yml -- playwright test test1"
  },
  "dependencies": {
    "scroll-to-bottomjs": "^1.1.0"
  }
}

And here is our first test:

// e2e/test1.spec.ts
import { test, expect, type Page } from '@playwright/test';
const percySnapshot = require('@percy/playwright');
let scrollToBottom = require("scroll-to-bottomjs");
test.beforeEach(async ({ page }) => {
    await page.goto('https://css-tricks.com', { waitUntil: 'domcontentloaded' });
});
test.describe('Go to page', () => {
    test('Do screenshot', async ({page}) => {
        // lazy load fix
        await page.evaluate(scrollToBottom);
        // const currentScreen = +new Date();
        const currentScreen = '1';
        await percySnapshot(page, 'testing1_css-tricks.com' + '_' + currentScreen);
    });
});

Fixing Percy missing images

On websites that are using lazy-loading, the final screenshots can be missing images / photos. A simple solution is to use https://www.npmjs.com/package/scroll-to-bottomjs , which will scroll to the bottom and force the images to be loaded. Then the snapshot method is executed. Our example already has this solution implemented.

Percy configuration

To adjust screenshot settings, such as breakpoints, custom CSS or enabling javascript, we are going to use the percy.yml file placed in the main GIT directory.

version: 2
snapshot:
  widths:
    - 375
    - 1280
  min-height: 1024
  percy-css: |
    .site-header .main-nav>ul>li>a {
      color:inherit !important;
    }
discovery:
  disable-cache: true

Regarding browser selection, these can be disabled in the Percy dashboard settings:

Run playwright tests

In our example, we use NODE v16.15.1, but the earlier versions should also be working. To install dependencies, we will execute:

npm install

To execute playwright tests, the following command will be useful:

npx playwright test test1

(it’s omitting percy snapshots, just checking if our test passes successfully).

To make percy working, we need to define a Percy Token and add it as an environment variable, for example:

export PERCY_TOKEN=abcd1234

or

set PERCY_TOKEN=abcd1234

or PowerShell format:

$env:PERCY_TOKEN = "abcd1234"

The last step is to execute the following command:

npm run runpercy_test1

this will execute scripts defined in package.json

(  percy exec –config ./percy.yml — playwright test test )

Once executed, the new build is starting processing in percy.io :

In 55 seconds, the screenshots are completed (1 width breakpoint x 4 browsers):

Percy – hiding elements / components

To prevent false positives, Percy.io does some adjustments to the site, for example: stopping animations, ignoring 1px layout shifts or doing smart font rendering. However, there might be a situation when some dynamic site elements, such as an advertisement or an autoplayed video, will interfere with tests and always signal the “changes to be reviewed” alert.

To hide dynamic elements in Percy, we can use percy.yml and apply our custom CSS. By using the visibility: hidden property for an element, we should resolve the issue. The element will be hidden, but the space of the element will be preserved. An example:

percy-css: |
    #main .ads-container {
      visibility:hidden !important;
    }

Percy.io browser versions

At the moment of writing this article: June 29 2022 – percy.io offers 4 browser engines that will render screenshots:

• Safari 14.0
• Firefox 95.0
• Chrome 96.0
• Edge 98.0

In my opinion, it’s a bit disappointing. For example, currently percy.io uses Chrome version number 96.0, which was released on: November 16, 2021. Why is there no 103.0 version (currently in use)?

I’ve reached out to customer support and got the following answer: “we update the browser versions to a stable version as soon as they are available to test in a span of 90 days. “

Automate snapshot creation

Currently, our script needs to be executed on a local machine to run. It’s nice for one time testing, but not ideal for monitoring public websites. Let’s configure Jenkins Job that will run every day at 3:25 in the morning, and execute regression visual tests.

Jenkins is an automation server and offers a convenient solution to set up continuous integration in the project. This example will show gitlab and jenkins integration: jenkins will authenticate to a GitLab repository using an access token, set up docker environment using Node v.16, install dependencies and perform defined tests (one test in our situation). The results will be sent to the percy.io API and presented in the Percy Dashboard.

Jenkins configuration

Jenkins – new Item (Freestyle project)

Jenkins project configuration:

* Source Code Management – Git

Repository URL (use https:// ) -> https://gitlab.example.com/repo1/testing1.git

Credentials ( choose access token previously configured globally )

* Branches to build: */master

* Build Triggers -> Build periodically ( cron like time format )

25 3 * * 1-5

would next run at * 3:25:18 AM CEST.

* Build – Execute shell

Command (make sure to replace PERCY_TOKEN with real value):

export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"  # This loads nvm
# chose desired node version
VERSION="16.15.1"
# install node
nvm install ${VERSION}
nvm use ${VERSION}
export PERCY_TOKEN=abcd1234
npm install; npx playwright install; npm run runpercy_test1;

GitLab configuration

We use self-hosted Gitlab, but this step should be similar for other code repositories. We need to authorize Jenkins to access the GIT repo in Gitlab and generate an “access token for gitlab project”. Just go to Gitlab / Particular repo page / Settings – Access Tokens :

• Token name: some friendly name
• Select a role: Developer
• Select scopes: read_repository

Jenkins credentials

To save Gitlab access credentials in Jenkins, go to: Credentials / System – Global credentials – Add Credentials :

• Kind: Username with password
• Scope: Global
• Username: youtGitlabUsername
• Password: just generated a gitlab project access token

Jenkins percy automation

That’s it. Our e2e testing script will be triggered every day at 3:25AM and the results will be visible in the percy.io dashboard. The Scheduled build took 12 seconds to complete (on container docker02). Example Jenkins console output:

> runpercy_test1
> percy exec --config ./percy.yml -- playwright test test1
[[95mpercy[39m] Percy has started!
[[95mpercy[39m] Running "playwright test test1"
Running 1 test using 1 worker
[[95mpercy[39m] Snapshot taken: testing1_css-tricks.com_1
[1A[2K[1/1] [chromium] › test1.spec.ts:11:5 › Go to page › Do screenshot
[1A[2K
1 passed (4s)
To open last HTML report run:
npx playwright show-report
[[95mpercy[39m] Uploading 1 snapshot...
[[95mpercy[39m] Finalized build #32: [34mhttps://percy.io/abc/test1/builds/19999[39m
Finished: SUCCESS

Our results for https://css-tricks.com/ have been processed in 39 seconds.  There is a difference that should be reviewed! Good news – nothing to worry about – just a new text, next to the post, has been added (Updated on Jun 28, 2022). Let’s see the visual changes in the percy dashboard, Safari webkit browser selected with 1280px viewport:

Git custom workflow

Percy can be integrated with Self-Managed Gitlab to sync visual builds with Merge Requests. A properly configured Gitlab pipeline can: deploy changes to the public staging server, run percy tests and notify about the “detection of any visual changes”.

By clicking “Details”, you will be redirected to the Percy build to review and approve changes before accepting the Merge Request. More info in Percy documentation: https://docs.percy.io/docs/source-code-integrations and  https://docs.percy.io/docs/gitlab-self-managed

Summary

We recommend percy.io, it’s fast in processing screenshots and provides useful information for the Developers or the QA Team. The clever AI behind the percy algorithm and wide browsers support should improve the QA process and speed up development.

The free plan with 5000 screenshots should be enough to start your journey with screenshot regression testing. What is the definition of a singular screenshot? According to the documentation: “A screenshot is a rendering of a page or component in an individual browser and responsive width combination”. So, if you define 3 breakpoints and set up 4 browsers, one percySnapshot call will generate 12 screenshots. Current usage history can be found in the dashboard.

Source code

Working playwright test code can also be found in the GitHub repository. Just clone it, add the percy token via the command line and check it yourself! Percy AI visual regressions – playwright example ->  https://github.com/createit-dev/148-percy-ai-visual-regression-playwright-example

That’s it for today’s tutorial. Make sure to follow us for other tips and guidelines and don’t forget to subscribe to our newsletter.