Allure Report Selenium and TestNG Tutorial

Allure Report Selenium and TestNG integration will be explained in all detail in this article. In my last article, I explained how to integrate ExtentReports reporting framework with TestNG for shiny and beautiful test automation reports. Now, I will explain to you another popular test reporting framework that the Yandex QA Team develops – Allure Reports

I will add additional codes and configurations to the ExtentReports project. Thus, in this code, you can generate test reports for both Allure and ExtentReports! Let’s start to integrate Allure into our test automation projects! If you are using JUnit 5, you can check Allure and JUnit 5 integration article.

Step-1: Modify Maven pom.xml 

For Allure integration, you need to add the below maven dependencies into your pom.xml file.

Properties Section

<properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <aspectj.version>1.9.6</aspectj.version>
    <selenium-version>4.0.0-beta-3</selenium-version>
    <testng-version>7.4.0</testng-version>
    <ashot-version>1.5.4</ashot-version>
    <allure-testng-version>2.13.9</allure-testng-version>
    <log4j-version>2.14.1</log4j-version>
    <extentreports-version>5.0.8</extentreports-version>
    <maven-surefire-plugin-version>3.0.0-M5</maven-surefire-plugin-version>
</properties>

Allure Report TestNG Dependency

<dependency>
    <groupId>org.testng</groupId>
    <artifactId>testng</artifactId>
    <version>${testng-version}</version>
</dependency>

Build Section of pom.xml in Allure Report Project

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <configuration>
                <source>11</source> <!--For JAVA 8 use 1.8-->
                <target>11</target> <!--For JAVA 8 use 1.8-->
            </configuration>
        </plugin>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-surefire-plugin</artifactId>
            <version>${maven-surefire-plugin-version}</version>
            <configuration>
                <suiteXmlFiles>
                    <suiteXmlFile>TestNG.xml</suiteXmlFile>
                </suiteXmlFiles>
                <argLine>
                    -javaagent:"${settings.localRepository}/org/aspectj/aspectjweaver/${aspectj.version}/aspectjweaver-${aspectj.version}.jar"
                </argLine>
            </configuration>
            <dependencies>
                <dependency>
                    <groupId>org.aspectj</groupId>
                    <artifactId>aspectjweaver</artifactId>
                    <version>${aspectj.version}</version>
                </dependency>
            </dependencies>
        </plugin>
    </plugins>
</build>

and we are done! The pom.xml settings have been finished.

The final pom.xml is below:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>allurereportexample</groupId>
    <artifactId>allurereportexample</artifactId>
    <version>1.0-SNAPSHOT</version>
    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <aspectj.version>1.9.6</aspectj.version>
        <selenium-version>4.0.0-beta-3</selenium-version>
        <testng-version>7.4.0</testng-version>
        <ashot-version>1.5.4</ashot-version>
        <allure-testng-version>2.13.9</allure-testng-version>
        <log4j-version>2.14.1</log4j-version>
        <extentreports-version>5.0.8</extentreports-version>
        <maven-surefire-plugin-version>3.0.0-M5</maven-surefire-plugin-version>
    </properties>
    <dependencies>
        <dependency>
            <groupId>org.seleniumhq.selenium</groupId>
            <artifactId>selenium-java</artifactId>
            <version>${selenium-version}</version>
        </dependency>
        <dependency>
            <groupId>org.testng</groupId>
            <artifactId>testng</artifactId>
            <version>${testng-version}</version>
        </dependency>
        <dependency>
            <groupId>ru.yandex.qatools.ashot</groupId>
            <artifactId>ashot</artifactId>
            <version>${ashot-version}</version>
        </dependency>
        <dependency>
            <groupId>io.qameta.allure</groupId>
            <artifactId>allure-testng</artifactId>
            <version>${allure-testng-version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-core</artifactId>
            <version>${log4j-version}</version>
        </dependency>
        <dependency>
            <groupId>com.aventstack</groupId>
            <artifactId>extentreports</artifactId>
            <version>${extentreports-version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-api</artifactId>
            <version>2.14.1</version>
            <scope>test</scope>
        </dependency>
    </dependencies>
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>11</source> <!--For JAVA 8 use 1.8-->
                    <target>11</target> <!--For JAVA 8 use 1.8-->
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>${maven-surefire-plugin-version}</version>
                <configuration>
                    <suiteXmlFiles>
                        <suiteXmlFile>TestNG.xml</suiteXmlFile>
                    </suiteXmlFiles>
                    <argLine>
                        -javaagent:"${settings.localRepository}/org/aspectj/aspectjweaver/${aspectj.version}/aspectjweaver-${aspectj.version}.jar"
                    </argLine>
                </configuration>
                <dependencies>
                    <dependency>
                        <groupId>org.aspectj</groupId>
                        <artifactId>aspectjweaver</artifactId>
                        <version>${aspectj.version}</version>
                    </dependency>
                </dependencies>
            </plugin>
        </plugins>
    </build>
</project>

Step-2: Configure Allure Report Features and Modify Test Codes

I will explain all features based on the below screenshot. I modified my test file to make the report HTML file more understandable and informative.

allure annotations

Feature-1: Display Name in Allure Report

To make our test report more understandable, I used the description property of @test annotation.

@Test (priority = 0, description=”Invalid Login Scenario with wrong username and password.”)

Also, you can add an additional description with @Description annotation:

@Description(“Test Description: Login test with wrong username and wrong password.”)

The result will be shown as below:

Feature-2: Steps in Allure Report

Steps are test actions in our test scenarios. They can be used for any testing scenario. Thus, we need to define steps in a generic place in our test project. To define a step, we need to use @Step annotation. In our project, steps are defined in our page classes. LogintoN11, verifyLoginUserName, verifyLoginPassword, all of these methods are our test steps. Therefore, we need to add @Step annotation above these methods. Here, we can also pass parameters with {} syntax. For example, in the first method, {0} is first parameter – username, {1} is second parameter – password.

LoginPage Steps:

allure step definitions

HomePage Steps:

steps

Here are the results. They will look at the report as in the below screenshot.

steps on reports

Feature-3: Attachments in Allure Report

We can add attachments to our reports by using @Attachment annotation. It can return String, byte [], etc. For example, if we want to attach a screenshot, we should return byte[]. Also, I need to add @Listeners({ TestListener.class }) declaration at the top of the test class.

allure attachment annotation

In TestListener class, I wrote two attachment methods for string attachment and screenshot attachment.

@Attachment annotation

And, I called those methods when a test failed as shown below.

saving screenshot in allure reporting

The result will be like this:

Allure Reports

Feature-4: Links in Allure Report

You can integrate your defect tracking system and test management tool with allure by using @Link annotation, as shown below. [1]

import io.qameta.allure.Link;
import io.qameta.allure.Issue;
import io.qameta.allure.TmsLink;

@Link("https://example.org")
@Link(name = "allure", type = "mylink")
public void testSomething() {
     ...
}

@Issue("123")
@Issue("432")
public void testSomething() {
     ...
}

@TmsLink("test-1")
@TmsLink("test-2")
public void testSomething() {
     ...
}

To specify the link pattern, you can use the system property in the following format: allure.link.my-link-type.pattern=https://example.org/custom/{}/path. Allure will replace {} placeholders with the value specified in the annotation. For example:

allure.link.mylink.pattern=https://example.org/mylink/{}
allure.link.issue.pattern=https://example.org/issue/{}
allure.link.tms.pattern=https://example.org/tms/{}

I will not use this feature in my test, so I cannot show you any results.

Feature-5: Severity in Allure Report

We can order tests by severity by using @Severity annotation. I used this feature in tests, as shown below.

@Severity annotation

and the result will be like that.

Severity on the test automation report

Feature-6: Behaviour-Driven Reporting (Features and Stories)

We can group tests with @Epic, @Feature, and @Stories annotations.

Epic Feature Story definitions

Here is the report result in Behaviors Section.

behaviors

and that’s all. :) Now, we should run the test and generate the report.

Step-3: Run the Test and Generate Allure Report

You can run the test with the maven command. In order to do this in IntelliJ, first, you should click configurations.

configuration

Select maven, write the maven “clean test” command as shown below, and then click OK.

mvn clean test

Now, we can run the code by clicking the green run icon.

test results

Now, it is time to generate the report!

To generate a report, we should install the Allure command-line interpreter.

  1. Download the latest version as a zip archive from bintray.
  2. Then, click the Files tab and then download the .zip file for windows. For Linux, you can download .tgz file. For Mac, use brew to install allure.
  3. Unpack the archive to the allure-commandline directory.
  4. Navigate to the bin directory.
  5. Add allure to system PATH.

allure installation on windows

If you are using MAC, then you can install allure with the below Brew command.

brew install allure

and finally, open a command prompt screen, go to the project directory, and write the below command!

allure serve allure-results

allure-results

and you will see the beautiful Allure Test Report as shown below.

Allure Report Dashboard

test automation report dashboard

Categories in Allure Report

categories

Suites in Allure Report

suites

Graphs in Allure Report

graphs

Timeline in Allure Report

timeline

Behaviors in Allure Report

Packages in Allure Reports

packages

and we integrate the Allure test framework with our automation project! :)

GitHub Project

https://github.com/swtestacademy/TestNGAllureReport 

References

[1] https://docs.qameta.io/allure/2.0/#_testng

See you in the next article!
Onur Baskirt

94 thoughts on “Allure Report Selenium and TestNG Tutorial”

  1. Hi Onur,
    Is this framework built on both Allure and ExtentReport?
    Can you just use Allure but still use the annotations and tags you used in ExtentReport?

    Reply
  2. HI Onur, that’s a helpful article for a starter of Allure Reporting. I was also interested in using ReportPortal reports ,but now where I was able to get a good tutorials to start with, I was able to set up report portal API, but unable to get some idea how to integrate it to my existing TestNG project.

    Please suggest me where I can get a step by step tutorial to start with,It would be more helpful if it by you guys.

    Regards,
    Amarnath

    Reply
    • Hi Amar, at work I can not do technical work. I am doing generally managerial stuff. :/ When I will hire a test automation engineer or find a suitable time, I am planning to integrate Report Portal with TestNG and publish that work. I will keep you updated. ;)

      Reply
    • I don’t understand “offline” meaning for Allure. You can use maven and it handles everything. It automatically downloads all required libraries. You don’t need to do an extra operation for .jars. All annotations worked in my example.

      Reply
  3. Can you please point to a link or article where we can generate Allure Report using Jenkins PipeLine , i.e., run my tests using Jenkins PipeLine and then generate the Allure Report using the same Jenkins Environment.

    Reply
  4. Thanks Onur for your response.
    I was saying about /htmlelements a qatool used by Yandex. AShot is also a good one but not complicated and advanced as Htmlelement is my thought.

    Reply
  5. Thank you onur, screenshot code structure was bit tricky to embed in my structure but once embedded, it worked like charm..

    Reply
  6. I have a problem to compile my project….

    [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test (default-test) on project nimitz-teste: Execution default-test of goal org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test failed: There was an error in the forked process
    [ERROR] org.testng.TestNGException:
    [ERROR]
    [ERROR] Cannot instantiate class org.testng.ITestListener
    [ERROR] at org.testng.internal.ClassHelper.newInstance(ClassHelper.java:71)
    [ERROR] at org.testng.internal.DefaultListenerFactory.createListener(DefaultListenerFactory.java:13)
    [ERROR] at org.testng.TestRunner.initListeners(TestRunner.java:315)
    [ERROR] at org.testng.TestRunner.init(TestRunner.java:274)
    [ERROR] at org.testng.TestRunner.init(TestRunner.java:241)
    [ERROR] at org.testng.TestRunner.(TestRunner.java:167)
    [ERROR] at org.testng.SuiteRunner$DefaultTestRunnerFactory.newTestRunner(SuiteRunner.java:663)
    [ERROR] at org.testng.SuiteRunner.init(SuiteRunner.java:260)
    [ERROR] at org.testng.SuiteRunner.(SuiteRunner.java:198)
    [ERROR] at org.testng.TestNG.createSuiteRunner(TestNG.java:1295)
    [ERROR] at org.testng.TestNG.createSuiteRunners(TestNG.java:1273)
    [ERROR] at org.testng.TestNG.runSuitesLocally(TestNG.java:1128)
    [ERROR] at org.testng.TestNG.runSuites(TestNG.java:1049)
    [ERROR] at org.testng.TestNG.run(TestNG.java:1017)
    [ERROR] at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:281)
    [ERROR] at org.apache.maven.surefire.testng.TestNGXmlTestSuite.execute(TestNGXmlTestSuite.java:75)
    [ERROR] at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:121)
    [ERROR] at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:290)
    [ERROR] at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:242)
    [ERROR] at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:121)
    [ERROR] Caused by: java.lang.InstantiationException: org.testng.ITestListener
    [ERROR] at java.lang.Class.newInstance(Class.java:427)
    [ERROR] at org.testng.internal.ClassHelper.newInstance(ClassHelper.java:69)
    [ERROR] … 19 more
    [ERROR] Caused by: java.lang.NoSuchMethodException: org.testng.ITestListener.()
    [ERROR] at java.lang.Class.getConstructor0(Class.java:3082)
    [ERROR] at java.lang.Class.newInstance(Class.java:412)
    [ERROR] … 20 more
    [ERROR]
    [ERROR] -> [Help 1]
    [ERROR]
    [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
    [ERROR] Re-run Maven using the -X switch to enable full debug logging.
    [ERROR]
    [ERROR] For more information about the errors and possible solutions, please read the following articles:
    [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException

    Reply
  7. Do you know by chance if there’s a way to configure Allure report to show only limited information on it’s main page? For example only the status circle or “Categories” pane?

    Reply
  8. Hi, I got report displayed correctly with above steps.
    But when tried to build jenkins item, the test body is missing in the report.
    Have no idea about this issue.
    Could you give some advice?

    Reply
  9. Thank you for this helpful content. I have prepared everything well but in the end, I get “NaN” and “unknown” screen on my Allure report screen. Couldn’t find a solution. Do you have an idea why it is happening?

    Reply
      • Im unable to create allure report due to jdk version error can you plz tell me why it is comming eventhough jdk version is correctly displays , also in system variable also set correct

        C:\Users\shital.pawar>java -version
        java version “1.8.0_202”
        Java(TM) SE Runtime Environment (build 1.8.0_202-b08)
        Java HotSpot(TM) 64-Bit Server VM (build 25.202-b08, mixed mode)

        C:\Users\shital.pawar>mvn -version
        The JAVA_HOME environment variable is not defined correctly,
        this environment variable is needed to run this program.

        C:\Users\shital.pawar>allure -version

        ERROR: JAVA_HOME is set to an invalid directory: C:\Program Files\Java\jdk1.8.0_291

        Please set the JAVA_HOME variable in your environment to match the
        location of your Java installation.

        Reply
  10. Hello Onur, thank you very much for your guides.
    I faced with some issue, im using TestNG, Cucumber with parallel execution for multiple mobile devices and i would like to use Allure as a reporter. So for now i have two options:
    1. Cucumber plugin for Allure.
    After my tests passed, report generates with whole information according to cucumber structure (feature name -> scenario name -> steps). But in this case there is only one report. Allure gets random device and generates report for that device only. In other words cucumber doesnt know about my other tests (which run in parallel).
    2. Testng plugin for Allure
    After passing tests allure generates reports for each device (thread) and everything is fine. But in this case, i testng doesnt know about cucumber steps, and there are no information about steps. So the structure is: feature name -> test name (it could be named as a cucumber scenario) and nothing else.
    My question is: is there any trick to display cucumber steps in allure report while using testng?
    Thank you in advance!

    Reply
  11. Thank you for your reply, i already have done it with @Step annotation. Now im thinking how to pass scenario names in, for example, @Story annotation. But as far as i know there is no way to pass dynamic values into annotations, so it might be impossible.

    Reply
  12. Hi Onur, I am not able to see the Test Steps in my Allure Report. I have defined the steps at page level with @Step(“Test Step”) annotation and also at the test class level after @Test annotation.

    Reply
    • Would you change maven surefire plugin version to 3.0.0-M3. I hope it will help. In our current project we are using this version without any problem. We are seeing Allure results. Please keep me updated if you will not solve your problem.

      Reply
  13. gradlew allureServe leaves server running. Is it possible to automatically stop the server after report is opened in browser?

    Reply
  14. can i enable allure report in data driven framework with technology Java, Selenium, TestNG, Log4J, Maven & DataBase? If yes please provide sample code for that in detail.

    Reply
  15. Hi Onur,

    How should I save my allure reports for future use,I want to save them and open anytime.If I stop running server I couldn’t able to view them.Can you guide on this please.

    Thanks inadvance.

    Reply
  16. Hi , I see this sub grouping in suites in one of your pictures https://imgur.com/a/3nKpHOb
    I am using allure reporter with selenium, webdriverio and mocha combination with javascript language. I would like to reflect the folder structure on disk in allure reports too. Like folder/test_file/test_cases like hierarchy in allure reports for suites/subsuites/test cases listed under that. https://imgur.com/a/Iq0aaTN

    Reply
  17. Hi, Onur. I built a project using Allure TestNg and successfully generated the report but the test body is missing. The allure component version I used is 2.13.2. I have stuck on this issue by several week. could you give me some advice on this.

    Reply
    • Thanks for your kind words. I will also try to update the articles. If you see any missing parts or errors, please let me know. I have started to update the articles and I am doing one-by-one thus soon or later I will update this one as well with latest tool versions.

      Reply
  18. Seems like I have a problem with @Steps and other annotations, they are not visible in report. Can you help me with advice please. Here is my pom file:

    11
    11
    11
    src/test/resources/testing.xml
    UTF-8
    1.9.6
    2.13.9
    7.4.0
    3.0.0-M5

    org.testng
    testng
    ${testng.version}

    io.qameta.allure
    allure-testng
    ${allure.version}

    org.slf4j
    slf4j-simple
    2.0.0-alpha1

    org.apache.maven.plugins
    maven-surefire-plugin
    ${maven-surefire-plugin}

    -Dfile.encoding=UTF-8
    -javaagent:”${settings.localRepository}/org/aspectj/aspectjweaver/${aspectj.version}/aspectjweaver-${aspectj.version}.jar”

    ${project.build.directory}/allure-results

    org.aspectj
    aspectjweaver
    ${aspectj.version}

    org.apache.maven.plugins
    maven-compiler-plugin
    3.1

    ${java.version}
    ${java.version}

    true

    io.qameta.allure
    allure-maven
    2.10.0

    Reply
    • Website refactor style a bit in my pom.
      11</maven.compiler.source
      11</maven.compiler.target
      11</java.version
      src/test/resources/testing.xml</testing.suiteXmlFile
      UTF-8</project.build.sourceEncoding
      1.9.6</aspectj.version
      2.13.9</allure.version
      7.4.0</testng.version
      3.0.0-M5</maven-surefire-plugin

      Reply
      • Yes the theme or wordpress is re-processing these XML files as I see. I will try to pull the code from github and re-run and reproduce the error and get back to you whenever I will have time for it but in the mean time you can play with testng and the allure versions. Maybe it is a library issue.

        Reply
  19. Hi Thanks for sharing such good stuff. It’s realy helpfull.
    One queation here, Observe 3rd screenshot(black screen – Run log window) of step no: 3 – In that the 2 logs are showing “Run 1: Pass” and “Run 2: Name of Test case etc…”.
    My queation is why it’s showing like that Run 1 and Run 2. I’ve implimented Retry login in framework and if test case fail it’s retry x number of time and at the end it’s showing like same as screenshot shown here. “Run 1: Pass, Run 2: Pass, Run 3: Error(Test case name etc)”? Please share ur thought on? Thanks!

    Reply
    • Hi Mahendra, I think it is the internal logging of TestNG, when I have time, I will re-try these tests with the latest version of TestNG. I do think that we may override these default messages but I need to check it hands-on.

      Reply
  20. Hi, thanks for share it. Do you know how to customize allure widgets? I just need to have a graph with all my test results ordered by feature/area. I need to use de folder name of each test inside my project, they are already ordered. Any customization documentation will be great! Thanks!

    Reply
  21. Hi,

    Thanks for this explanation. I have a question. When I run my test cases parallelly, the allure report is not generated. Can you help me on how to do it? And you are using allure-testng library. Can I know for what it is used.

    Reply
  22. I wrote the tests for the allure report according to your pom.xml everything seems correct and the tests run. In the console it says that the screenshot is taken, but it is not in the allure report. If you can help, I will be very grateful.

    Reply
  23. Hi.
    Faced with issue when run my test with TestNG runner.
    After run, my allure-results folder are generates on the project root directory not in /target directory..
    My pom.xml contains following items:
    2.16.1
    2.16.1
    2.10.0
    1.9.6
    2.16.1
    Also I have a allure.properties file with allure.results.directory=target/allure-results
    But it doesn’t help..
    So, any suggestions? What it can be and how to fix it??
    Will appreciate any help.

    Reply

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.