Visual Testing with Applitools, Appium, and Amazon Device Farm

Visual UI testing is more than just testing your app on Desktop browsers and Mobile emulators. In fact, you can do more with Visual UI testing to run your tests over physical mobile devices.

Visual UI testing compares the visually-rendered output of an application against itself in older iterations. Users call this type of test version checking. Some users apply visual testing for cross-browser tests. They run the same software version across different target devices/operating systems/browsers/viewports. For either purpose, we need a testing solution that has high accuracy, speed, and works with a range of browsers and devices. For these reasons, we chose Applitools.

Running your Visual UI testing across physical devices means having to set up your own local environment to run the tests. Imagine the number of devices, screen resolutions, operating systems, and computers you’d need! It would be frustratingly boring, expensive, and extremely time-consuming.

This is where Amazon Device Farm comes into play. This powerful service can build a testing environment. It uses physical mobile devices to run your tests! All you do is upload your tests to Amazon, specify the devices you want, and it will take it from there!

In one of my recent articles, How Visual UI Testing can speed up DevOps flow I showed how you can configure a CD/CI service to run your Visual UI tests. The end result would be the same, whether you are running your tests locally, or via such services. Once the tests run, you can always check the results over the Applitools Test Manager Dashboard.

In this article, I will show you how you can run your Visual UI tests, whether you’ve written them for your mobile or web app, on real physical mobile devices in the cloud. For this, I will be employing Applitools, Appium, and Amazon Device Farm.

Amazon Device Farm for mobile visual testing

Amazon Device Farm is a mobile app testing platform that helps developers automatically test their apps on hundreds of real devices in minutes.

When it comes to testing your app over mobile devices, the choices are numerous. Amazon helps to build a “Device Farm” on behalf of the developers and testers, hence the name.

Here are some of the major advantages and features for using this service:

• Java/Kotlin for Android.
• Swift for iOS.
• PhoneGap.
• Xamarin.
• Unity.
• And web apps built for mobile browsers.

Amazon Device Farm supports a number of test runners. This includes Appium Java JUnit, Appium Python, Appium Ruby, and Appium Java TestNG. Back in January 2019, Amazon announced support for the Appium Node.js test runner. This means you can build your tests with Selenium Webdriver, for instance, and have it run on top of Amazon Device Farm.

Now that you have an idea about Amazon Device Farm, let’s move on, and discover the Appium automation testing framework.

Selenium Webdriver for browser app automation

Selenium WebDriver is a browser automation framework that allows a developer to write commands, and send them to the browser. It offers a set of clients with a variety of programming languages (Java, JavaScript, Ruby, Python, PHP and others).

Figure 1 below shows the Selenium WebDriver architecture:

Figure 1: Selenium WebDriver Architecture

Selenium WebDriver architecture consists of:

Selenium 4 is obseleting the JSONWP in favor of the new W3C WebDriver standard.

Here’s a quick tutorial on using and learning Selenium WebDriver.

With that brief overview of Selenium WebDriver, let’s move on and explore Appium.

Appium for mobile app automation

Appium is an open-source tool to automate Mobile app testing. It’s a cross-platform that supports both OS (Android and iOS) test scripts. It is tested on simulators (iOS), emulators (Android) and real devices (iOS, Android).

It’s an HTTP Server written in Node.js that creates and handles WebDriver sessions. When you install Appium, you are actually installing the Appium Server. It follows the same approach as the Selenium WebDriver, which receives HTTP requests from the Client Libraries in JSON format with the help of JSONWP. It then handles those HTTP Requests in different ways. That’s why you can make use of Selenium WebDriver language bindings, client libraries and infrastructure to connect to the Appium Server.

Instead of connecting a Selenium WebDriver to a specific browser WebDriver, you will be connecting it to the Appium Server. Appium uses an extension of the JSONWP called the Mobile JSON Wire Protocol (MJSONWP) to support the automation of testing for native and hybrid mobile apps.

It supports the same Selenium WebDriver clients with a variety of multiple programming languages such as Java, JavaScript, Ruby, Python, PHP and others.

Being a Node.js HTTP Server, it works in a client-server architecture. Figure 2 below depicts the Appium Client-Server Architecture model:

Figure 2: Appium Server Architecture

Appium architecture consists of:

The results of the test session are then communicated back to the Appium Server, and back to the Client in the form of logs, using the Mobile JSONWP.

Now that you are well equipped with knowledge for Selenium WebDriver and Appium, let’s go to the demo section of this article.

Demo

In this section, we will write a Visual UI test script to test a Web page. We will run the tests over an Android device both locally and on Amazon Device Farm.

I will be using both Selenium WebDeriver and Appium to write the test script.

Prerequisites

Before you can start writing and running the test script, you have to make sure you have the following components installed and ready to be used on your computer:

• Java installed
• JAVA_HOME environment variable is set to the Java SDK path
• Node.js installed

Assuming you are working on a MacOS computer, you can verify the above installations by running the following bash commands:

echo $JAVA_HOME // this should print the Java SDK path
node -v // this should print the version of Node.js installed
npm -v // this should print the version of the Node Package Manager installed

Component Installations

For this demo we need to install Appium Server, Android Studio / SDK and finally make sure to have a few environment variables properly set.

Let’s start by installing Appium Server. Run the following command to install Appium Server locally on your computer.

npm install -g appium

The command installs the Appium NPM package globally on your computer. To verify the installation, run the command:

appium -v // this should print the Appium version

Now let’s install Android Studio / SDK so that you can run the test script on an emulator or real device. You could install the Android SDK only but then you have to do additional advanced steps to properly configure the Android environment on your computer. I highly recommend installing the Android Studio as it makes your life easier.

Download the Android Studio executable. Follow the steps below to install locally on your computer:

Notice the location where the Android SDK was installed. It’s /Users/{User Account}/Library/Android/sdk.

Wait until the download and installation is complete. That’s all!

Because I want to run the test script locally over an Android emulator, let’s add one.

Open the Android Studio app:

Click the Configure icon:

Select the AVD Manager menu item.

Click the + Create Virtual Device button.

Locate and click the Pixel XL device then hit Next.

Locate the Q release and click the Download link.

Read and accept the Terms and Conditions then hit Next.

The Android 10, also known as Q release, starts downloading.

Once the installation is complete, click the Next button to continue setting up an Android device emulator.

The installation is complete. Grab the AVD Name as you will use it later on in the test script, and hit Finish.

Finally, we need to make sure the following environment variables are set on your computer. Open the ~/.bash_profile file, and add the following environment variables:

APPLITOOLS_API_KEY={Get the Applitools API Key from Applitools Test Manager}
export APPLITOOLS_API_KEY
ANDROID_HOME=/Users/{Use your account name here}/Library/Android/sdk
export ANDROID_HOME
ANDROID_HOME_TOOLS=$ANDROID_HOME/tools
export ANDROID_HOME_TOOLS
ANDROID_HOME_TOOLS_BIN=$ANDROID_HOME_TOOLS/bin
export ANDROID_HOME_TOOLS_BIN
ANDROID_HOME_PLATFORM=$ANDROID_HOME/platform-tools
export ANDROID_HOME_PLATFORM
APPIUM_ENV="Local"
export APPIUM_ENV

Finally, add the above environment variables to the $PATH as follows:

export $PATH=$PATH:$ANDROID_HOME:$ANDROID_HOME_TOOLS:$ANDROID_HOME_TOOLS_BIN:$ANDROID_HOME_PLATFORM

One last major component that you need to download, and have on your computer, is the ChromeDriver. Navigate to the Appium ChromeDriver website, and download the latest workable ChromeDriver release for Appium. Once downloaded, make sure to move the file to the location: /usr/local/bin/chromedriver

That’s it for the installations! Let’s move on and explore the Visual UI test script in depth.

Run the Visual UI Test Script locally

You can find the source code demo of this article on this GitHub repo.

Let’s explore the main test script in this repo.

The test script starts by importing the selenium-webdriver NPM package.

It imports a bunch of objects from the @applitools/eyes-selenium NPM package.

It constructs a BatchInfo object used by Applitools API.

const batchInfo = new BatchInfo("AWS Device Farm"); batchInfo.id = process.env.BATCH_ID batchInfo.setSequenceName('AWS Device Farm Batches');

It then creates the Eyes object that we will use to interact with the Applitools API.

// Initialize the eyes SDK
let eyes = new Eyes();
eyes.setApiKey(process.env.APPLITOOLS_API_KEY);
eyes.setLogHandler(new FileLogHandler(true));
eyes.setForceFullPageScreenshot(true)
eyes.setStitchMode(StitchMode.CSS)
eyes.setHideScrollbars(true)
eyes.setBatch(batchInfo);

It’s so important to set the Applitools API key at this stage. Otherwise, you won’t be able to run this test. The code above also directs the Applitools API logs to a File located at the root of the project under the name of eyes.log.

Next, we define the device capabilities that we are going to send to Appium.

const capabilities = {
 platformName: "Android",
 deviceName: "Android Emulator",
 automationName: "UiAutomator2",
 browserName: 'Chrome',
 waitforTimeout: 30000,
 commandTimeout: 30000,
 };
 if (process.env.APPIUM_ENV === "Local") {
 capabilities["avd"] = 'Pixel_XL_API_29';
 }

We are using an Android emulator to run our test script over a Chrome browser with the help of the UIAutomator 2 library.

We need to set the avd capability only when running this test script locally. For this property, grab the AVD ID of the Android Device Emulator we set above.

Now, we create and build a new WebDriver object by specifying the Appium Server local URL and the device capabilities as:

const LOCAL_APPIUM = "http://127.0.0.1:4723/wd/hub";
let driver = new webdriver
 .Builder()
 .usingServer(LOCAL_APPIUM)
 .withCapabilities(capabilities)
 .build();

Appium is configured to listen on Port 4723 under the path of /wd/hub.

The rest of the script is usual Applitools business. In brief, the script:

• Opens a new Applitools test session
• Sends a command to navigate the browser to https://us.vuejs.org/
• Grabs the page title and displays it on screen.
• Clicks the Burger Button to expand the menu on a Mobile device.
• Finds the Location section on the page.
• Finally, it prints the H2 text of the Location section.

Notice that the script asserts two Eyes SDK Snapshots. The first captures the home page of the website, while the second captures the Location section.

Finally, some important cleanup is happening to close the WebDriver and Eyes SDK sessions.

Open the package.json file, and locate the two scripts there:

"appium": "appium --chromedriver-executable /usr/local/bin/chromedriver --log ./appium.log",
"test": "node appium.js"

The first runs and starts the Appium Server, and the second to run the test script.

Let’s first run the Appium server by issuing this command:

npm run-script appium

Then, once Appium is running, let’s run the test script by issuing this command:

npm run-script test

Verify Test Results on Applitools Test Manager

Login to the Applitools Test Manager located at: https://applitools.com/users/login

You will see the following test results:

The two snapshots have been recorded!

Run the Visual UI Test Script on Amazon Device Farm

Now that the test runs locally, let’s run it on Amazon Device Farm. Start by creating a new account on Amazon Web Service website.

Login to your AWS account on this page: https://console.aws.amazon.com/devicefarm

Create a new project by following the steps below:

• Select the Mobile Device Project and name your project Appium.
• Click the Create project button.

• Select the HTML5 option since we are testing a Web Page on a mobile device.
• Assign your test run a name.
• Click the Next step button.

• Select the Appium Node.js test runner
• Upload your tests packaged in a zip file.

Let’s package our app in a zip file in order to upload it at this step.

Switch back to the code editor, open a command window, and run the following:

npm install

This command is essential to make sure all the NPM package dependencies for this app are installed.

npm install -g npm-bundle

The command above installs the npm-bundle NPM package globally on your machine.

Then, run the command to package and bundle your app:

npm-bundle

The command bundles and packages your app files, and folders, including the node_modules folder.

The output of this step creates the file with the .tgz extension.

The final step before uploading is to compress the file by running the command:

zip -r appium-aws.zip *.tgz

Name the file whatever you wish.

Now you can upload the .zip file to Amazon Device Farm.

Once the file uploads, scroll down the page to edit the .yaml file of this test run like so:

• Make sure you insert your Applitools API Key as shown in the diagram.
• Add the node appium.js command to run your test script.
• Click the Save Testspec file.

• Give this new pool a name.
• Pick up the selected Android Devices. You may select others too.
• Click the Save device pool button.

• Now, you can see the new device pool selected with the devices listed.
• Click the Next step button.

• The test run starts!
• Select the test run listed.

Verify Test Results on Applitools Test Manager

Switch back to the Applitools Test Manager, and verify the results of this second run via Amazon Device Farm.

As expected, we get exactly the same results as running the test script locally.

Conclusion

Given the massive integrations that Applitools offers with its rich SDKs, we saw how easily and quickly we can run our Visual UI tests in the cloud using the Amazon Device Farm service. This service, and similar services, enrich the Visual regression testing ecosystem, and make perfect sense when performing them.

For More Information

• Request an Applitools Demo
• Sign up for a free Applitool account
• Learning Appium Visual Testing on TAU
• Cross-browser Testing With Cypress.io
• How Visual UI Testing can speed up DevOps flow

Originally published at https://applitools.com on June 18, 2020.