Test Automation Reporting with Allure and TestNG

Hi all, in my last article I explained how to integrate ExtentReports reporting framework with TestNG for shiny and beautiful test automation reports [link]. In this article, I will explain you another popular test reporting framework which is developed by Yandex QA Team – Allure.  I will add additional codes and configurations on ExtentReports project. Thus, in this code, you can both generate test reports both Allure and ExtentReports! 🙂 Let’s start to integrate Allure into our test automation projects! If you are using JUnit 5, you can check Allure and JUnit 5 integration in this article.

Step-1: Modify Maven pom.xml 

For Allure integration, you need to add below maven dependencies into your pom.xml file.

Properties Section

Allure TestNG Dependency

Build Section

and we are done! The pom.xml settings have been finished.

The final pom.xml is below:

Step-2: Configure Allure Features and Modify Test Codes

I will explain all features based on the below picture. I did some modifications on my test file to make the report HTML file more understandable and informative.

Feature-1: Display Name

In order to make our test report more understandable, I used description property of @test annotation.

@Test (priority = 0, description=”Invalid Login Scenario with wrong username and password.”)

Also, you can add an additional description with @Description annotation:

@Description(“Test Description: Login test with wrong username and wrong password.”)

The result will be shown as below:

Feature-2: Steps

Steps are test actions in our test scenarios. They can be used for any testing scenario. Thus, we need to define steps in a generic place in our test project. In order to define a step, we need to use @Step annotation. In our project, steps are defined at our page classes. LogintoN11, verifyLoginUserName, verifyLoginPassword, all of these methods are our test steps. Therefore, we need to add @Step annotation above these methods as shown below. Here, we can also pass parameters with {} syntax. For example, at first method, {0} is first parameter – username, {1} is second parameter – password.

LoginPage Steps:

HomePage Steps:

Here are the results. They will look on the report as like below image.

Feature-3: Attachments

We can add attachments to our reports by using @Attachment annotation. It can return String, byte [], etc. For example, if we want to attach a screenshot we should return byte[]. Also, I need to add @Listeners({ TestListener.class }) declaration at the top of the test class.

In TestListener class, I wrote two attachment method for string attachment and screenshot attachment.

And, I called those methods when a test failed as shown below.

The result will be like below in the report.

Feature-4: Links

You can integrate your defect tracking system and test management tool with allure by using @Link annotation as shown below. [1]

I will not use this feature in my test so I cannot show you any result for this.

Feature-5: Severity

We can order test by severity by using @Severity annotation. I used this feature in tests as shown below.

and the result will be like that.

Feature-6: Behaviour Driven Reporting (Features and Stories)

We can group tests with @Epic, @Feature, and @Stories annotations.

Here is the report result in Behaviors Section.

and that’s all. 🙂 Now, we should run the test and generate the report.

Project Code is here: https://github.com/swtestacademy/TestNGAllureReport 

Step-3: Run the Test and Generate Allure Report

You can run the test with maven command. In order to do this in IntelliJ first, you should click configurations.

Select maven, and write the maven “clean test” command as shown below and then click OK.

Now, we can run the code by clicking the green run icon.

Now, it is time to generate the report!

In order to generate a report, we should install Allure command-line interpreter.

1. Download the latest version as a zip archive from bintray.
2. Then, click the Files tab and then download the .zip file for windows. For Linux, you can download .tgz file. For Mac use brew to install allure.
3. Unpack the archive to allure-commandline directory.
4. Navigate to bin directory.
5. Add allure to system PATH.

If you are using MAC, then you can install allure with below Brew command.

and finally, open a command prompt screen, go to the project directory, and write below command!

allure serve allure-results

and, you will see the beautiful Allure Test Report as shown below.

Dashboard

Categories

Suites

Graphs

Timeline

Behaviors

Packages

and we integrate Allure test framework with our automation project! 🙂

Project Code is here: https://github.com/swtestacademy/TestNGAllureReport 

Selenium Webdriver Tutorial Series

See you in next article!
-Onur

References

[1] https://docs.qameta.io/allure/2.0/#_testng