Automate Your PDF Testing

Automated testing of PDFs has traditionally been a challenging task in test automation. Bitmap imaging, the legacy state-of-the-art approach, has proven unreliable. Most automation teams have focused on their end-to-end user tests, except PDF tests. Instead, they have relegated PDF validation to manual testers. And, alas, manual testing is prone to error.

In this article, we will review the requirements of PDF testing. We will also cover approaches to automating PDF tests using Applitools.

Why PDF Testing?

Organizations cannot own all possible UI failures. When users generate their own PDFs via the local operating system, they may encounter errors. But, when organizations generate receipts, account statements, and other documents for customer download, the organizations clearly own problems with those. A nd, in regulated industries, PDF problems can incur regulator wrath in addition to customer frustration. Thus, testing the generated outputs is mandatory from both quality and legal perspectives.

Organizations use Applitools to uncover problems in their transactional or customer-related PDF documents

What to Automate?

Consider an application producing customer letters using a PDF template. Various sections of the PDF are dynamically updated with the customer data. A test for PDF needs to verify that both the content and layout of the output document are correct.

When testing for layout, the document should be fully formed with:

  • the specific sections present
  • in the right location and
  • in the right order.

When testing for content, testers need to ensure that:

A layout failure can impact the processing of the documents by downstream systems. Testers need to be able to check content on a page or on locations on all or selected pages.

How traditionally organizations did PDF Testing?

Usually, organizations take an approach of validating the data using API testing and finally using solutions such as PDF boxes to test them on a page. However, most organizations rely on manual testing to validate the full document output. As organizations generate increasing numbers of electronic documents, they see the scope of effort needed for comprehensive testing. These organizations test PDF a sample of their documents manually.

Application of Visual AI in testing PDF

In the case of a PDF, Applitools lets users inspect the entire document, or just selected pages. The scope of comparison matches the needs of the test. Users can target specific sections of a PDF for testing and ignore sections that are not relevant to a test. And, armed with the Applitools layout algorithm, users can validate a PDF layout even if the internal text has changed.

PDF Testing solution

Look at how the previous example can be tested using Applitools PDF Tester. Configure the test job as an XML. Add the content validations as test assertions. You can create the XML manually, or you can build it programmatically using a script. Once completed, execute the PDF test using the Applitools PDF Test application. You can run the test from the command line using any batch process.

Following are the results:

The utility reports all the assertions in the PDF document and reports the result as ‘Passed’ or ‘Failed’. In addition, the utility tests the fully formatted output document against a baseline and reports any differences discovered.

Logging into Applitools dashboard we can review all the differences spotted by AI:

Here, AI is highlighting all the content, size, color, positioning, and font differences. Engineers can validate the differences, ensuring document accuracy prior to publication. We can further instruct AI to specifically test or ignore sections of the page by using annotations.

Conclusion

Originally published at https://applitools.com on March 5, 2021.

Deliver visually perfect web and mobile apps with AI-powered end-to-end visual testing & monitoring. Get started, free: http://bit.ly/ApplitoolsFree

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store