In our recent series of articles, published on the Atlassian Community, we’ve compared 3 popular testing apps for Jira – QAlity Plus, Xray, and Zephyr Squad. We’ve discussed the similarities and differences in test writing, test cycles, test executions, and reporting, showcasing the capabilities of each product. Today, we would like to summarize the series and provide an overview of each app’s offer.
Test writing
The differences between test writing-related features in the 3 apps are predominantly the matter of interface. Most of these differences do not have any significant impact on the core functionalities such as test creation, editing, cloning, or deleting, which are present in each solution, and work in a similar manner. There are, however, some distinct characteristics worth mentioning in order to paint the whole picture of how each app approaches the test writing topic.
Key points of comparison:
Below, you will find a list of the crucial insights discussed in the complete comparison article.
Creating tests
Creating tests by choosing a test issue type: all 3 apps
Creating tests directly in an existing issue: QAlity Plus
Creating test steps
Adding test steps at the end of the test steps list: all 3 apps
Adding test steps in between already created steps, on a chosen position: Xray
Viewing options
Offering two different viewing options, one more detailed and the other simplified: all 3 apps
Collapsing or expanding the test steps view: QAlity Plus, Xray
Adding additional columns and customizing the elements that are to be shown: Zephyr Squad
Editing
All elements edited inline: QAlity Plus
Type and definition edited inline, test steps edited in an edit mode: Xray
All elements edited in an edit mode: Zephyr Squad
Reordering
Reordering steps by dragging and dropping: all 3 apps
Manually setting the position for a test step: Xray
Cloning and deleting
Cloning and deleting feature available in test issue menu: all 3 apps
Cloning and deleting test steps by clicking on action icons: Xray, Zephyr Squad
If you need a detailed comparison, we’ve covered the topic in this article: Test writing in QAlity Plus, Xray and Zephyr Squad
Test cycles
Test organization introduces more differences between QAlity Plus, Xray, and Zephyr Squad. Each app offers its own unique system of grouping test cases. Depending on the user’s needs, different approaches to test cycles can turn out to be the most suitable.
Key points of comparison:
Below, you will find a short description of the QAlity Plus, Xray, and Zephyr Squad’s offer, highlighting the most distinct points for each.
Overview
QAlity Plus
Test Cycle Overview page provides a comprehensive view of your test cycles, offering essential details such as project, version it is assigned to, status of test executions, creation date, and due date. It also allows filtering of test cycles based on both project and version criteria.
Xray
Tests can be organized into Test Plans, instead of test cycles. Test Plans provide an insight into the most crucial information on the related tests: a list of tests, their executions, dataset, latest status as well as the overall execution status. Additionally, you can group tests in Test Sets or Test Repositories.
Zephyr Squad
Test Cycle Summary page displays the details about your test cycles, depending on the selected version of the project. Basic information includes status, summary, start date, end date, and more. You can also modify the columns displayed in your Test Cycle Summary.
Creating test cycles
Creating test cycles from the test cycles page: QAlity Plus, Zephyr Squad
Creating a test plan by choosing a test plan issue type: Xray
Reordering test cases within test cycles
Reordering test cases by dragging and dropping: all 3 apps
Manually setting the position for a test case: Xray
If you need a detailed comparison, we’ve covered the topic in this article: Test Cycles in QAlity Plus, Xray and Zephyr Squad.
Folders and Repository
Organizing test items is also possible in folders. They introduce more flexibility into your test management, allowing you to group test items according to your needs. Folders can be used to segregate items related to different modules of your application. Another common use case is to organize test cases in folders to easily add them to test cycles later. All 3 apps provide folder-related functionality. Each one of them also gives you access to a Repository, where you can view and manage your items.
Key points of comparison:
Below, you will find a list of the crucial characteristics and differences between the apps’ offer.
Creating folders: all 3 apps
Adding to folders test cases: QAlity Plus
Adding to folders tests and preconditions: Xray
Adding to folders test cases, test plans and test cycles: Zephyr Squad
Adding to a folder items from multiple projects: QAlity Plus
Adding one item to multiple folders: QAlity Plus
Creating sub-folders: Xray, Zephyr Squad
Test Case Repository: QAlity Plus, Zephyr Squad
Test Repository: Xray
Test Execution
The core part of testing is, of course, test execution. The structure and interface of this feature can significantly impact the comfort and efficiency of the tester’s work.
Key points of comparison:
Below, you will find a short summary of the QAlity Plus, Xray, and Zephyr Squad’s offer, with the key differences listed.
Starting the execution process
Ad hoc or as a part of Test Cycle or Test Plan: all 3 apps
By creating a test execution issue type: Xray
General info on the execution page
QAlity Plus: execution assignee, description, comments
Xray: assignee, executed by, versions, revision, dates, timer, environments, findings, test details
Zephyr Squad: status, assignee, summary, description, when last performed and by whom, defects, comments
Assigning users
QAlity Plus: on the Test Cycle page or within a specific Test Case Execution
Xray: creating separate test execution issues for each tester or assigning runs within test execution issue based on specific user type fields
Zephyr Squad: on the Test page or Test Cycles page
Recording results
Available statuses:
QAlity Plus: Unexecuted, In Progress, Passed, Failed, Blocked
Xray: Todo, Executing, Pass, Fail + Custom
Zephyr Squad: Unexecuted, WIP, Pass, Fail, Blocked + Custom
Overall execution status
Execution status is what informs the user about the current result of testing. Different apps approach naming and updating statuses differently. Each solution is also characterized by its own overall execution status update requirements.
Overall execution status:
QAlity Plus: If all tests are passed, overall execution status will automatically be set to passed as well.
Xray: Overall execution status is automatically adjusted to the current status of test executions.
Zephyr Squad: Overall execution status does not update automatically, and can only be changed manually.
Reporting defects
QAlity Plus: linking or creating bugs directly from the Test Execution page; bug description generated automatically (the executed test case name, execution date, link to the execution, and test case steps)
Xray: linking or creating bugs in the Findings section of your Test Execution; linking or creating bugs for specific Test Steps
Zephyr Squad: linking or creating bugs for either the Test Execution or the specific Test Steps
Adding attachments
Adding attachments to specific Test Steps: all 3 apps
Adding attachments to the whole Test Execution: Xray, Zephyr Squad
Execution history
QAlity Plus: available within the current Test Execution; includes information such as how many times the test case has been executed already, the date and status of the previous execution, and the date when the test case last failed; consists of two tabs – Bugs reported and Steps failed
Xray: available in the Test Runs section of the Test issue page
Zephyr Squad: provides information such as status changes, attachment actions (add/update/remove), assignee modifications, comment actions (add/update/remove), and custom field adjustments
Test case versioning
QAlity Plus: during test execution, the current version is locked and cannot be modified
Xray: creating and handling different versions of a test within the same Test issue
Zephyr Squad: no test case versioning
If you need a detailed comparison, we’ve covered the topic in this article: Test Executions in QAlity Plus, Xray and Zephyr Squad
Reporting
There would be no use for testing without functional reporting features in place, providing the user with an insight into the results of test execution and the overall condition of the tested software.
Reports exporting features in each app:
Below, you will find a rundown of the QAlity Plus, Xray, and Zephyr Squad’s offer, focusing on the reporting options available in each app.
Test Execution Reports
QAlity Plus: report of all test executions for a chosen period of time; two additional views available – status chart and assignee chart; can be exported to an .XLSX file
Xray: in-built reports for particular pages (Tests List Report, Test Sets List Report, Test Executions List Report, Test Plans List Report, Test Runs List Report, Test Plans Metrics Report); can be exported to a CSV file
Zephyr Squad: Test Metrics (Daily Test Execution Progress by status, Test Executions by Test Cycle, Test Executions by Tester, Number of executions per day, List of executions with their details); reports for specific projects (Test Execution Chart, Test Execution Burndown Chart, Top Defects Impacting Testing Report), no exporting available
Coverage & Traceability Reports
QAlity Plus: Coverage and Traceability combined into one report, no exporting available
Xray: two separate reports for Coverage and Traceability; Traceability Report can be exported to a CSV file
Zephyr Squad: Traceability Matrix, with forward and backward reports available; can be exported to an Excel file
If you need a detailed comparison, we’ve covered the topic in this article: Reporting in QAlity Plus, Xray and Zephyr Squad
Cloud Pricing
Here are the monthly cloud prices for each of the three apps, shown in three examples – for up to 10 users, for 50 users, and for 1000 users:
Summary
Testing plays a significant role in any development process. For that reason, there are countless testing apps for Jira available on the Atlassian Marketplace. Examining the differences between each solution can be a laborious and time-consuming task. To make it less daunting, we’ve explored the offer of 3 popular testing apps – QAlity Plus, Xray, and Zephyr Squad, and compiled the unique features and differentiating characteristics of each.
Our aim was to construct a comprehensive overview of the available options, to provide the fuller picture of their approach to the topic of testing.
The comparison series revealed both small and big differences between the apps. The details were already discussed extensively in the previous articles of the series. Generally speaking, each app has its own points of focus and can answer the needs of different users.
QAlity Plus is the simplest out of 3. Being true to the “less is more” phrase, it puts the attention on intuitiveness, aiming to interact with the user in a way that is easy to understand. At the same time, it provides all fundamental testing features that other 2 apps also have. What QAlity Plus lacks, is some of the advanced functionalities that would make the app a lot more complex and less user-friendly. When it comes to pricing, QAlity Plus is the most cost-efficient option, being 2 to 3 times less expensive than Xray and Zephyr Squad.
Xray, in terms of complexity, is the opposite of QAlity Plus. It offers advanced, multi-layered features that require a lot of time and focus to learn, but can be very useful in organizing tests and executing them. Xray puts a lot of effort into creating a unique system. It makes the app less intuitive than QAlity Plus, but enables more complex operations. As for the pricing, Xray is the only app out of 3 that requires payment even for a 10 users instance. The overall pricing is higher than QAlity Plus, but lower than Zephyr Squad.
Zephyr Squad, compared to the other 2 apps, allows more customization. The user can modify the view according to their needs by adding or removing columns that are to be displayed. This interactive approach gives the user a chance to decide what information is the most crucial to them, and what should stay hidden to not obstruct the view. Zephyr Squad is the most costly option out of the 3 apps compared in this series.
Each app has its own strengths and weaknesses. We hope that this comparison series helped you to determine which of the 3 solutions meets your requirements, and suits your team best.
