Iris 2.0 Overview

Summary:

Iris 2.0 is a significant leap from our 2018 efforts. It will enable faster test writing, scale for the future, and most importantly, position Iris as an open-source visual testing platform that goes beyond desktop Firefox.

Targeted release in early May, 2019.

(Jump to feature breakdown and owners)

Architecture:

A port to Python 3 allows us to finally be a modern project, and take advantage of the latest Python features. In this port, we can eliminate technical debt from the previous iteration and build a cleaner, better-designed project.

We will rewrite the project to modularize Firefox as a component, setting the stage for extensibility that will allow Iris to be used to automate any desktop application. We will document this feature with an SDK and make the process as simple as possible.

We will replace custom code with standard libraries whenever possible, including adoption of pytest [link], which will attract contributors with previous experience in that test framework. With the use of pytest, we will also convert our test library to a simpler format that scales better.

We hope to port the mozdownload and mozbase libraries to Python 3, which will also help other teams who depend on them.

Test capabilities:

We will finally make OCR a first-class citizen in Iris 2.0 by taking advantage of Tesseract 4. This is also improved by smarter region handling, which will assist in pattern search as well.

We will enhance core logic around keyboard and mouse handling to enhance test stability as well as to enable us to run in virtualized environments.

We would like to load assets directly from the Firefox codebase, to allow dynamic testing that matches the latest build of Firefox. We will load and convert graphical assets, which should minimize work when these assets change. We also would like to load strings, to assist in localization and to avoid the process of manual pattern creation for each locale.

Deployment:

A completely new Control Center will present Iris as a polished, professional tool, with a clean and streamlined design.

We will present a solution for running Iris in a batch mode, to trigger sequential runs, ultimately simplifying the process of running update tests.

We will explore better run reporting, via HTML formatting of email reports, and/or dynamic generation of Google docs.

We will research ways to reduce "oranges" or intermittent/false errors, via technical solutions and/or process improvements.

Team productivity:

We will explore further CI solutions by integrating Travis with AWS/Docker. We will also research integrating code quality tools such as SonarQube if they can make our work easier and/or more efficient.

Feature breakdown and ownership

• Improved project design and architecture
  • Port to Python 3 (Silviu, Andrei, Hutu, Matt)
  • Port mozdownload/mozbase to Python 3 (Hutu)
  • New project structure (Matt)
  • Extensibility (Matt, Silviu, Andrei)
  • Integrate pytest (Silviu, Matt)
• New test case format
  • Test case base class (Andrei)
  • Port test library (TBD)
• Keyboard and Mouse observers (Andrei, Roxana)
• OCR improvements (Hutu, Silviu)
• Smart region handling (Ioana, Silviu)
• Dynamic pattern loading (Roxana, Matt)
• Dynamic string loading (Roxana, Matt)
• Control Center
  • UI redesign (Paul, Kristin, Matt)
  • Control Center data (Matt, Silviu)
• CI improvements (AWS/Docker/SonarQube) (Roxana,Andrei)
• Batch run script (Hutu)
• Better reports (HTML, Google doc, pastebin) (Andrei, Ioana, Ionut, Silviu)
• Bugzilla integration (Ionut, Silviu)
• Better orange handling (TBD)