How To Run 372 Cross Browser Tests In Under 3 Minutes

Image for post
Image for post

How do you run cross browser testing today?

Applitools released a new product and it looks to be rather innovative. Focused on visual testing, Applitools first caught my eye with a 2015 demo which showed how their algorithms could detect the difference between a layout modification and a content modification on a webpage, quite the step up from pixel-perfect image diffing that other early visual UI testing tools were doing.

Their new product, Visual Grid, drastically improves the speed of running a suite of cross browser visual UI tests. Visual Grid improves speed and reduces cost for all teams using visual web testing, but I’m personally excited about the opportunity for promoting visual testing habits directly to frontend developers. Frontend developers can now get instant feedback for their changes across an array of desktop and mobile browsers. Applitools bets that if the total time of visual UI test suites can be reduced to seconds or minutes, they can be fit into the instant feedback cycle employed by modern developers while actively coding. There’s a huge difference between waiting an hour to find out if your commits broke something and waiting a minute. Retaining a developer’s flow-state boosts productivity, and reduces frustration.

Hands-On Experience

62 pages * 2 browsers * 3 screen sizes

= 372 images captured and verified in less than 3 minutes

Crossing the 1–3 minute mark for an entire cross-browser test suite took considerable technical effort on the part of Applitools, and Visual Grid takes a novel approach that, as far as I’m aware, nobody else is offering. The crux of their innovation is that a single browser is automated to crawl a site locally on the developer’s machine while a fleet of machines in their cloud take screenshots and compare them to past images.

The developer opens up Chrome locally and visits, the Visual Grid client then sends all of the HTML elements, images, text, etc currently displayed in Chrome to the Visual Grid Cloud. The Applitools browsers in the cloud load the same HTML assets and take a screenshot, effectively testing the layout and rendering in every browser and screen size selected. The traditional way of gathering these comparison screenshots is to use a Selenium cloud provider, spinning up an entire automation session on each browser for testing, navigating to, loading the site, and taking a screenshot.

Avoid The Headaches Of Cross-Browser Testing

The actual user actions being automated on each webpage are only performed locally on a single browser, while screenshots are rendered and taken in the cloud. You shouldn’t have to pay for Applitools’ remote browsers to click on checkboxes and fill in forms, with Visual Grid you only use their cloud resources for rendering the screens generated by the automation script running locally.

When running automated tests on your site, are you concerned with testing whether Firefox correctly scrolls a combo box, or Safari can put placeholder text in a textbox? Of course not, what we’re all testing for is that the site looks right in every browser, and all our controls are visible. Visual Grid cuts out the overhead of testing browser actions, allowing us to run visual regression tests extremely quickly.

For those familiar with the current paradigm on using Applitools along with a cloud of automated browsers, here are three cases which demonstrate the difference when using Visual Grid:

Case Study 1: Random Number Generator

All four screenshots will show the same random number which was displayed on the developer’s local Chrome instance. The Visual Grid browsers did not visit the web page in question. Instead, a full copy of the HTML elements, styling, images, etc (the HTML Snapshot), is sent to each of the Visual Grid browsers to render.

Case Study 2: IP Address

As with Case Study 1, all four screenshots will show the IP address of the developer’s local computer. It’s interesting to note that the site’s server will only log a single HTTP request emanating from the local Chrome instance, the four Visual Grid browsers never send requests to the site.

Case Study 3: Javascript Code Which Detects Browser Type

The Firefox screenshots contain the buggy behavior. The HTML snapshots sent to Visual Grid show the same exact elements on the local developer’s screen at the time of the snapshot. Javascript on the page does not get rerun for each of the Visual Grid Browsers.

As Case Study 3 demonstrates, the speed we gain from using Visual Grid comes at a cost. There exists a class of bugs which Visual Grid will not catch: cross browser incompatibility of the javascript run on the page. Applitools does not argue that one should replace full end-to-end browser automation tests, but points out that the majority of cross-browser issues have to do with rendering and layout. In the modern day of large frontend frameworks, such as React and Angular, cross-browser bugs tend to be layout and style issues. The framework developers are the ones who mainly need to worry about browser javascript incompatibilities.

Inserting Visual Grid Into The Testing Pyramid

Image for post
Image for post

As we move up the pyramid, tests take longer to run and are less reliable but exercise the entire software stack more rigorously. Visual UI tests using Applitools Visual Grid are much faster and less flakey than full browser automation tests, and this translates to cost savings for users. I love how Visual Grid raises the bar for tests which can provide feedback to frontend developers while they are actively coding.

The only drawback is that Visual Grid won’t catch rare issues related to cross-browser _javascript_ execution and race conditions, for these you’ll still want some form of full browser automation. But you don’t need to test 100% of your functionality with full end-to-end automation tests. Don’t forget how the pyramid tapers towards the top: when you move many of your current end-to-end tests to use Visual Grid, you make available more time to test specific browser differences, while the Visual Grid tests runs cheaply on a greater diversity of browsers.

For more detail, including a great explanation of the Testing Pyramid and more detail on how Visual Grid is implemented, see this Webinar by Applitools Architect Gil Tayar. I support Gil’s opinion that in order for a software organization to produce a high-quality product and be able to iterate quickly, developers need to write and run their own tests. Applitools Visual Grid enables frontend developers to work more closely with their visual UI testing.

Want To Know More About Applitools?

Other reading for you:

  1. Applitools Appium Native Java Tutorial
  2. How To Do Cross-browser Visual Testing with Selenium
  3. Applitools FAQ — Cross Browser Testing by Yarden Naveh
  4. Applitools Cross Browser Testing Blog Posts

Deliver visually perfect web and mobile apps with AI-powered end-to-end visual testing & monitoring. Get started, free:

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store