How To Modernize Your Functional Testing

Screen Shot 2019 10 22 at 3.25.41 PM
  • The existence of the logo at top
  • The page title “Login Form”
  • The title and icons of the “Username” and “Password” text boxes
  • The existence of the “Sign in” button and the “Remember Me” checkbox — with the correct filler text
  • The Twitter and Facebook icons below
Screen Shot 2019 10 22 at 3.55.24 PM
  1. Identifiers might change, while the underlying content does not.
  2. Some identifiers have different structures, such as additional data that was not previously tested.
  3. Path-based identifiers depend on the relative location on a page, which can change from version to version.

Implications For Testing

Let’s walk through all the tests for just this page. We covered the negative condition for hitting “Sign In” without entering a username or password. Next, we must verify the error messages and other elements if one, or the other, field has data (the response messages may differ). Also, we need to handle the error condition where the username or password is incorrect.

Screen Shot 2019 10 22 at 4.31.51 PM

Visual Validation with Visual AI — A Better Way

What would happen if QA had a better way of validating the output — not just line by line and assertion by assertion? What if QA could take a snapshot of the entire page after an action and compare that with the previous instance?

  1. Drive behavior (with your favorite test runner)
  2. Take a screenshot

Handling Code Changes With Legacy Functional Test

The big benefit of visual validation with Visual AI comes from comparing new code with new features to old code.

Screen Shot 2019 10 23 at 11.26.45 AM
  1. Your existing test misses the broken logo (-1). The logo at the top no longer links to the proper icon file. Did you check for the logo? If you checked to see that the reference icon file was identical, your code misses the fact that the response file is a broken image.
  2. Your existing test misses the text overlap of the alert and the “Login Form” text (-1). The error message now overlaps the Login Form page title. You miss this error because the text remains identical — though their relative positions change
  3. Your existing test catches the new text in the username and password boxes (+2). Your test code correctly identifies that there is new prompt text in the name and password boxes and registers an error. So, your test shows two errors.
  4. Your existing test misses the new feature (-1). The new box with “Terms and Conditions” has no test. It is a new feature, and you need to code a test for the new feature.
Screen Shot 2019 10 23 at 11.26.45 AM 1
  • Your existing test gives you two false alarms that the Twitter and Facebook links are broken. Those links at the bottom used Xpath locators — which got changed by the new feature. Because the locators changed, these now show up as errors — false positives — that must be fixed to make the test code work again.

Handling Visual Changes with Visual AI

With Visual AI in Applitools, you actually capture all the important changes and correctly identify visual elements that remain unchanged, even if the underlying page structure is different.

Screen Shot 2019 10 23 at 11.38.40 AM
  • 1 — Change — the broken logo on top
  • 2 — Change — The login form and alert overlap
  • 3 — Change — The fact that the alert text has moved from version to version (the reason for the text overlap)
  • 4 — Change/Change — The changes to the Username and Password box text
  • 5 — Change — the new feature of the Terms and Conditions text
  • 6 — No Change — Twitter and Facebook logos remain unmoved (no false positives)

Comparing Visual AI and Legacy Functional Code

With Visual AI, you no longer have to look at the screen and determine which code changes to make. You are asked to either accept the change as part of the new baseline or reject the change as an error.

Screen Shot 2019 10 23 at 11.54.37 AM
  • The real bug — the broken logo — can only be uncovered by manual testing. Once it is discovered, a tester needs to determine what code will find the broken representation. Typically, this can take 15 minutes (more or less). And you need to inform the developers of the bug.
  • The visual bug — the text overlap — can only be uncovered by manual testing. Once the bug is discovered, the tester needs to determine what code will find the overlap and add the appropriate test (e.g. a CSS check). This could take 15 minutes (more or less) to add the test code. And, you need to inform the developers of the bug.
  • The intentionally changed placeholder text for Username and Password text boxes need to be recoded, as they are flagged as errors. This could take 10 minutes (more or less).
  • The new feature can only be identified by manual validation or by the developer. This test needs to be added (perhaps 5 minutes of coding). You may want to ask the developers about a better way to find out about the new features.
  • The false positive errors around the Twitter and Facebook logos need to be resolved. The Xpath code needs to be inspected and updated. This could take 15 minutes (more or less)
  • You find the broken logo by running visual validation. No additional code or manual work needed. Incremental time: zero seconds. Alert developers of the error: 2 seconds. Alerting in Applitools can send a bug notification to developers when rejecting a difference.
  • Visual validation uncovers the text overlap and moved alert text. Incremental time: zero secondsAlert developers of the error: 2 seconds. Alerting in Applitools can send a bug notification to developers when rejecting a difference.
  • Visual validation identifies the new text in the Username and Password text boxes. Visual validation workflow lets you accept the visual change as the new baseline (2 seconds per change — or 4 seconds total)
  • You uncover the new feature with no incremental change or new test code, and you accept the visual change as the new baseline (2 seconds).
  • The Twitter and Facebook logos don’t show up as differences — so you have no work to do (0 seconds)

Let’s Get Real

I would think that a productivity gain of 360X might appear unreasonable. So did Raja. When he went through the real-world examples for writing and maintaining tests, he came up with a more reasonable-looking table.

Screen Shot 2019 10 23 at 12.15.46 PM
Screen Shot 2019 10 23 at 12.25.42 PM

Chapter 1 Summary

Raja summarizes the chapter by pointing out that visual validation with Visual AI requires only the following:

  1. Take action
  2. Take a screenshot in Visual AI
  3. Compare the screenshot with the baseline in Visual AI
Screen Shot 2019 10 23 at 12.27.50 PM

For More Information

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Applitools

Applitools

Deliver visually perfect web and mobile apps with AI-powered end-to-end visual testing & monitoring. Get started, free: http://bit.ly/ApplitoolsFree