Allure Report Selenium and TestNG Tutorial [2021 Update]

Allure Report Selenium and TestNG integration will be explained in all details in this article. In my last article, I explained how to integrate ExtentReports reporting framework with TestNG for shiny and beautiful test automation reports. In this article, I will explain to you another popular test reporting framework that is developed by Yandex QA Team – Allure Reports.  I will add additional codes and configurations to the ExtentReports project. Thus, in this code, you can generate test reports for both Allure and ExtentReports! :) Let’s start to integrate Allure into our test automation projects! If you are using JUnit 5, you can check Allure and JUnit 5 integration in this article.

Step-1: Modify Maven pom.xml 

For Allure integration, you need to add the below maven dependencies into your pom.xml file.

Properties Section

<properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <aspectj.version>1.9.6</aspectj.version>
    <selenium-version>4.0.0-beta-3</selenium-version>
    <testng-version>7.4.0</testng-version>
    <ashot-version>1.5.4</ashot-version>
    <allure-testng-version>2.13.9</allure-testng-version>
    <log4j-version>2.14.1</log4j-version>
    <extentreports-version>5.0.8</extentreports-version>
    <maven-surefire-plugin-version>3.0.0-M5</maven-surefire-plugin-version>
</properties>

Allure Report TestNG Dependency

<dependency>
    <groupId>org.testng</groupId>
    <artifactId>testng</artifactId>
    <version>${testng-version}</version>
</dependency>

Build Section of pom.xml in Allure Report Project

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <configuration>
                <source>11</source> <!--For JAVA 8 use 1.8-->
                <target>11</target> <!--For JAVA 8 use 1.8-->
            </configuration>
        </plugin>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-surefire-plugin</artifactId>
            <version>${maven-surefire-plugin-version}</version>
            <configuration>
                <suiteXmlFiles>
                    <suiteXmlFile>TestNG.xml</suiteXmlFile>
                </suiteXmlFiles>
                <argLine>
                    -javaagent:"${settings.localRepository}/org/aspectj/aspectjweaver/${aspectj.version}/aspectjweaver-${aspectj.version}.jar"
                </argLine>
            </configuration>
            <dependencies>
                <dependency>
                    <groupId>org.aspectj</groupId>
                    <artifactId>aspectjweaver</artifactId>
                    <version>${aspectj.version}</version>
                </dependency>
            </dependencies>
        </plugin>
    </plugins>
</build>

and we are done! The pom.xml settings have been finished.

The final pom.xml is below:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>allurereportexample</groupId>
    <artifactId>allurereportexample</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <aspectj.version>1.9.6</aspectj.version>
        <selenium-version>4.0.0-beta-3</selenium-version>
        <testng-version>7.4.0</testng-version>
        <ashot-version>1.5.4</ashot-version>
        <allure-testng-version>2.13.9</allure-testng-version>
        <log4j-version>2.14.1</log4j-version>
        <extentreports-version>5.0.8</extentreports-version>
        <maven-surefire-plugin-version>3.0.0-M5</maven-surefire-plugin-version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.seleniumhq.selenium</groupId>
            <artifactId>selenium-java</artifactId>
            <version>${selenium-version}</version>
        </dependency>

        <dependency>
            <groupId>org.testng</groupId>
            <artifactId>testng</artifactId>
            <version>${testng-version}</version>
        </dependency>

        <dependency>
            <groupId>ru.yandex.qatools.ashot</groupId>
            <artifactId>ashot</artifactId>
            <version>${ashot-version}</version>
        </dependency>

        <dependency>
            <groupId>io.qameta.allure</groupId>
            <artifactId>allure-testng</artifactId>
            <version>${allure-testng-version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-core</artifactId>
            <version>${log4j-version}</version>
        </dependency>

        <dependency>
            <groupId>com.aventstack</groupId>
            <artifactId>extentreports</artifactId>
            <version>${extentreports-version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-api</artifactId>
            <version>2.14.1</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>11</source> <!--For JAVA 8 use 1.8-->
                    <target>11</target> <!--For JAVA 8 use 1.8-->
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>${maven-surefire-plugin-version}</version>
                <configuration>
                    <suiteXmlFiles>
                        <suiteXmlFile>TestNG.xml</suiteXmlFile>
                    </suiteXmlFiles>
                    <argLine>
                        -javaagent:"${settings.localRepository}/org/aspectj/aspectjweaver/${aspectj.version}/aspectjweaver-${aspectj.version}.jar"
                    </argLine>
                </configuration>
                <dependencies>
                    <dependency>
                        <groupId>org.aspectj</groupId>
                        <artifactId>aspectjweaver</artifactId>
                        <version>${aspectj.version}</version>
                    </dependency>
                </dependencies>
            </plugin>
        </plugins>
    </build>
</project>

Step-2: Configure Allure Report Features and Modify Test Codes

I will explain all features based on the below picture. I did some modifications to my test file to make the report HTML file more understandable and informative.

allure annotations

Feature-1: Display Name in Allure Report

In order to make our test report more understandable, I used the description property of @test annotation.

@Test (priority = 0, description=”Invalid Login Scenario with wrong username and password.”)

Also, you can add an additional description with @Description annotation:

@Description(“Test Description: Login test with wrong username and wrong password.”)

The result will be shown as below:

Feature-2: Steps in Allure Report

Steps are test actions in our test scenarios. They can be used for any testing scenario. Thus, we need to define steps in a generic place in our test project. In order to define a step, we need to use @Step annotation. In our project, steps are defined at our page classes. LogintoN11, verifyLoginUserName, verifyLoginPassword, all of these methods are our test steps. Therefore, we need to add @Step annotation above these methods as shown below. Here, we can also pass parameters with {} syntax. For example, at first method, {0} is first parameter – username, {1} is second parameter – password.

LoginPage Steps:

allure step definitions

HomePage Steps:

steps

Here are the results. They will look at the report as like the below image.

steps on reports

Feature-3: Attachments in Allure Report

We can add attachments to our reports by using @Attachment annotation. It can return String, byte [], etc. For example, if we want to attach a screenshot we should return byte[]. Also, I need to add @Listeners({ TestListener.class }) declaration at the top of the test class.

allure attachment annotation

In TestListener class, I wrote two attachment methods for string attachment and screenshot attachment.

@Attachment annotation

And, I called those methods when a test failed as shown below.

saving screenshot in allure reporting

The result will be like this:

Allure Reports

Feature-4: Links in Allure Report

You can integrate your defect tracking system and test management tool with allure by using @Link annotation as shown below. [1]

import io.qameta.allure.Link;
import io.qameta.allure.Issue;
import io.qameta.allure.TmsLink;

@Link("https://example.org")
@Link(name = "allure", type = "mylink")
public void testSomething() {
     ...
}

@Issue("123")
@Issue("432")
public void testSomething() {
     ...
}

@TmsLink("test-1")
@TmsLink("test-2")
public void testSomething() {
     ...
}

In order to specify the link pattern you can use the system property in the following format: allure.link.my-link-type.pattern=https://example.org/custom/{}/path. Allure will replace {} placeholders with the value specified in the annotation. For example:

allure.link.mylink.pattern=https://example.org/mylink/{}
allure.link.issue.pattern=https://example.org/issue/{}
allure.link.tms.pattern=https://example.org/tms/{}

I will not use this feature in my test so I cannot show you any result for this.

Feature-5: Severity in Allure Report

We can order tests by severity by using @Severity annotation. I used this feature in tests as shown below.

@Severity annotation

and the result will be like that.

Severity on the test automation report

Feature-6: Behaviour Driven Reporting (Features and Stories)

We can group tests with @Epic, @Feature, and @Stories annotations.

Epic Feature Story definitions

Here is the report result in Behaviors Section.

behaviors

and that’s all. :) Now, we should run the test and generate the report.

Step-3: Run the Test and Generate Allure Report

You can run the test with the maven command. In order to do this in IntelliJ first, you should click configurations.

configuration

Select maven, and write the maven “clean test” command as shown below, and then click OK.

mvn clean test

Now, we can run the code by clicking the green run icon.

test results

Now, it is time to generate the report!

In order to generate a report, we should install the Allure command-line interpreter.

  1. Download the latest version as a zip archive from bintray.
  2. Then, click the Files tab and then download the .zip file for windows. For Linux, you can download .tgz file. For Mac use brew to install allure.
  3. Unpack the archive to the allure-commandline directory.
  4. Navigate to the bin directory.
  5. Add allure to system PATH.

allure installation on windows

If you are using MAC, then you can install allure with the below Brew command.

brew install allure

and finally, open a command prompt screen, go to the project directory, and write the below command!

allure serve allure-results 

allure-results

and, you will see the beautiful Allure Test Report as shown below.

Allure Report Dashboard

test automation report dashboard

Categories in Allure Report

categories

Suites in Allure Report

suites

Graphs in Allure Report

graphs

Timeline in Allure Report

timeline

Behaviors in Allure Report

Packages in Allure Reports

packages

and we integrate the Allure test framework with our automation project! :)

GitHub Project

https://github.com/swtestacademy/TestNGAllureReport 

References

[1] https://docs.qameta.io/allure/2.0/#_testng

See you in the next article!
Onur Baskirt

75 thoughts on “Allure Report Selenium and TestNG Tutorial [2021 Update]”

  1. Hi Onur,
    Is this framework built on both Allure and ExtentReport?
    Can you just use Allure but still use the annotations and tags you used in ExtentReport?

    Reply
  2. HI Onur, that’s a helpful article for a starter of Allure Reporting. I was also interested in using ReportPortal reports ,but now where I was able to get a good tutorials to start with, I was able to set up report portal API, but unable to get some idea how to integrate it to my existing TestNG project.

    Please suggest me where I can get a step by step tutorial to start with,It would be more helpful if it by you guys.

    Regards,
    Amarnath

    Reply
    • Hi Amar, at work I can not do technical work. I am doing generally managerial stuff. :/ When I will hire a test automation engineer or find a suitable time, I am planning to integrate Report Portal with TestNG and publish that work. I will keep you updated. ;)

      Reply
    • I don’t understand “offline” meaning for Allure. You can use maven and it handles everything. It automatically downloads all required libraries. You don’t need to do an extra operation for .jars. All annotations worked in my example.

      Reply
  3. Can you please point to a link or article where we can generate Allure Report using Jenkins PipeLine , i.e., run my tests using Jenkins PipeLine and then generate the Allure Report using the same Jenkins Environment.

    Reply
  4. Thanks Onur for your response.
    I was saying about /htmlelements a qatool used by Yandex. AShot is also a good one but not complicated and advanced as Htmlelement is my thought.

    Reply
  5. Thank you onur, screenshot code structure was bit tricky to embed in my structure but once embedded, it worked like charm..

    Reply
  6. I have a problem to compile my project….

    [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test (default-test) on project nimitz-teste: Execution default-test of goal org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test failed: There was an error in the forked process
    [ERROR] org.testng.TestNGException:
    [ERROR]
    [ERROR] Cannot instantiate class org.testng.ITestListener
    [ERROR] at org.testng.internal.ClassHelper.newInstance(ClassHelper.java:71)
    [ERROR] at org.testng.internal.DefaultListenerFactory.createListener(DefaultListenerFactory.java:13)
    [ERROR] at org.testng.TestRunner.initListeners(TestRunner.java:315)
    [ERROR] at org.testng.TestRunner.init(TestRunner.java:274)
    [ERROR] at org.testng.TestRunner.init(TestRunner.java:241)
    [ERROR] at org.testng.TestRunner.(TestRunner.java:167)
    [ERROR] at org.testng.SuiteRunner$DefaultTestRunnerFactory.newTestRunner(SuiteRunner.java:663)
    [ERROR] at org.testng.SuiteRunner.init(SuiteRunner.java:260)
    [ERROR] at org.testng.SuiteRunner.(SuiteRunner.java:198)
    [ERROR] at org.testng.TestNG.createSuiteRunner(TestNG.java:1295)
    [ERROR] at org.testng.TestNG.createSuiteRunners(TestNG.java:1273)
    [ERROR] at org.testng.TestNG.runSuitesLocally(TestNG.java:1128)
    [ERROR] at org.testng.TestNG.runSuites(TestNG.java:1049)
    [ERROR] at org.testng.TestNG.run(TestNG.java:1017)
    [ERROR] at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:281)
    [ERROR] at org.apache.maven.surefire.testng.TestNGXmlTestSuite.execute(TestNGXmlTestSuite.java:75)
    [ERROR] at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:121)
    [ERROR] at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:290)
    [ERROR] at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:242)
    [ERROR] at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:121)
    [ERROR] Caused by: java.lang.InstantiationException: org.testng.ITestListener
    [ERROR] at java.lang.Class.newInstance(Class.java:427)
    [ERROR] at org.testng.internal.ClassHelper.newInstance(ClassHelper.java:69)
    [ERROR] … 19 more
    [ERROR] Caused by: java.lang.NoSuchMethodException: org.testng.ITestListener.()
    [ERROR] at java.lang.Class.getConstructor0(Class.java:3082)
    [ERROR] at java.lang.Class.newInstance(Class.java:412)
    [ERROR] … 20 more
    [ERROR]
    [ERROR] -> [Help 1]
    [ERROR]
    [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
    [ERROR] Re-run Maven using the -X switch to enable full debug logging.
    [ERROR]
    [ERROR] For more information about the errors and possible solutions, please read the following articles:
    [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException

    Reply
  7. Do you know by chance if there’s a way to configure Allure report to show only limited information on it’s main page? For example only the status circle or “Categories” pane?

    Reply
  8. Hi, I got report displayed correctly with above steps.
    But when tried to build jenkins item, the test body is missing in the report.
    Have no idea about this issue.
    Could you give some advice?

    Reply
  9. Thank you for this helpful content. I have prepared everything well but in the end, I get “NaN” and “unknown” screen on my Allure report screen. Couldn’t find a solution. Do you have an idea why it is happening?

    Reply
  10. Hello Onur, thank you very much for your guides.
    I faced with some issue, im using TestNG, Cucumber with parallel execution for multiple mobile devices and i would like to use Allure as a reporter. So for now i have two options:
    1. Cucumber plugin for Allure.
    After my tests passed, report generates with whole information according to cucumber structure (feature name -> scenario name -> steps). But in this case there is only one report. Allure gets random device and generates report for that device only. In other words cucumber doesnt know about my other tests (which run in parallel).
    2. Testng plugin for Allure
    After passing tests allure generates reports for each device (thread) and everything is fine. But in this case, i testng doesnt know about cucumber steps, and there are no information about steps. So the structure is: feature name -> test name (it could be named as a cucumber scenario) and nothing else.
    My question is: is there any trick to display cucumber steps in allure report while using testng?
    Thank you in advance!

    Reply
  11. Thank you for your reply, i already have done it with @Step annotation. Now im thinking how to pass scenario names in, for example, @Story annotation. But as far as i know there is no way to pass dynamic values into annotations, so it might be impossible.

    Reply
  12. Hi Onur, I am not able to see the Test Steps in my Allure Report. I have defined the steps at page level with @Step(“Test Step”) annotation and also at the test class level after @Test annotation.

    Reply
    • Would you change maven surefire plugin version to 3.0.0-M3. I hope it will help. In our current project we are using this version without any problem. We are seeing Allure results. Please keep me updated if you will not solve your problem.

      Reply
  13. gradlew allureServe leaves server running. Is it possible to automatically stop the server after report is opened in browser?

    Reply
  14. can i enable allure report in data driven framework with technology Java, Selenium, TestNG, Log4J, Maven & DataBase? If yes please provide sample code for that in detail.

    Reply
  15. Hi Onur,

    How should I save my allure reports for future use,I want to save them and open anytime.If I stop running server I couldn’t able to view them.Can you guide on this please.

    Thanks inadvance.

    Reply
  16. Hi , I see this sub grouping in suites in one of your pictures https://imgur.com/a/3nKpHOb
    I am using allure reporter with selenium, webdriverio and mocha combination with javascript language. I would like to reflect the folder structure on disk in allure reports too. Like folder/test_file/test_cases like hierarchy in allure reports for suites/subsuites/test cases listed under that. https://imgur.com/a/Iq0aaTN

    Reply
  17. Hi, Onur. I built a project using Allure TestNg and successfully generated the report but the test body is missing. The allure component version I used is 2.13.2. I have stuck on this issue by several week. could you give me some advice on this.

    Reply
    • Thanks for your kind words. I will also try to update the articles. If you see any missing parts or errors, please let me know. I have started to update the articles and I am doing one-by-one thus soon or later I will update this one as well with latest tool versions.

      Reply

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.