Hi all, in my last article I explained how to integrate ExtentReports reporting framework with TestNG for shiny and beautiful test automation reports [link]. In this article, I will explain you another popular test reporting framework which is developed by Yandex QA Team – Allure. I will add additional codes and configurations on ExtentReports project. Thus, in this code, you can both generate test reports both Allure and ExtentReports! 🙂 Let’s start to integrate Allure into our test automation projects! If you are using JUnit 5, you can check Allure and JUnit 5 integration in this article.
Step-1: Modify Maven pom.xml
For Allure integration, you need to add below maven dependencies into your pom.xml file.
Properties Section
1 2 3 4 | <properties> <aspectj.version>1.8.10</aspectj.version> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> </properties> |
Allure TestNG Dependency
1 2 3 4 5 | <dependency> <groupId>io.qameta.allure</groupId> <artifactId>allure-testng</artifactId> <version>2.0-BETA19</version> </dependency> |
Build Section
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 | <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <configuration> <source>1.8</source> <target>1.8</target> </configuration> </plugin> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> <version>2.20</version> <configuration> <suiteXmlFiles> <suiteXmlFile>TestNG.xml</suiteXmlFile> </suiteXmlFiles> <argLine> -javaagent:"${settings.localRepository}/org/aspectj/aspectjweaver/${aspectj.version}/aspectjweaver-${aspectj.version}.jar" </argLine> </configuration> <dependencies> <dependency> <groupId>org.aspectj</groupId> <artifactId>aspectjweaver</artifactId> <version>${aspectj.version}</version> </dependency> </dependencies> </plugin> </plugins> </build> |
and we are done! The pom.xml settings have been finished.
Step-2: Configure Allure Features and Modify Test Codes
I will explain all features based on below picture. I did some modifications on my test file to make the report HTML file more understandable and informative.
Feature-1: Display Name
In order to make our test report more understandable, I used description property of @test annotation.
@Test (priority = 0, description=”Invalid Login Scenario with wrong username and password.”)
Also, you can add an additional description with @Description annotation:
@Description(“Test Description: Login test with wrong username and wrong password.”)
The result will be shown as below:
Feature-2: Steps
Steps are test actions in our test scenarios. They can be used for any testing scenario. Thus, we need to define steps in a generic place in our test project. In order to define a step, we need to use @Step annotation. In our project, steps are defined at our page classes. LogintoN11, verifyLoginUserName, verifyLoginPassword, all of this methods are our test steps. Therefore, we need to add @Step annotation above these methods as shown below. Here, we can also pass parameters with {} syntax. For example, at first method, {0} is first parameter – username, {1} is second parameter – password.
LoginPage Steps:
HomePage Steps:
Here are the results. They will look on the report as like below image.
Feature-3: Attachments
We can add attachments to our reports by using @Attachment annotation. It can return String, byte [], etc. For example, if we want to attach a screenshot we should return byte[]. Also, I need to add @Listeners({ TestListener.class }) declaration at the top of the test class.
In TestListener class, I wrote two attachment method for string attachment and screenshot attachment.
And, I called those methods when a test failed as shown below.
The result will be like below in the report.
Feature-4: Links
You can integrate your defect tracking system and test management tool with allure by using @Link annotation as shown below. [1]
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 | import io.qameta.allure.Link; import io.qameta.allure.Issue; import io.qameta.allure.TmsLink; @Link("https://example.org") @Link(name = "allure", type = "mylink") public void testSomething() { ... } @Issue("123") @Issue("432") public void testSomething() { ... } @TmsLink("test-1") @TmsLink("test-2") public void testSomething() { ... } |
In order to specify the link pattern you can use the system property in the following format: allure.link.my-link-type.pattern=https://example.org/custom/{}/path
. Allure will replace {}
placeholders with the value specified in the annotation. For example:
1 2 3 | allure.link.mylink.pattern=https://example.org/mylink/{} allure.link.issue.pattern=https://example.org/issue/{} allure.link.tms.pattern=https://example.org/tms/{} |
I will not use this feature in my test so I cannot show you any result for this.
Feature-5: Severity
We can order test by severity by using @Severity annotation. I used this feature in tests as shown below.
and the result will be like that.
Feature-6: Behaviour Driven Reporting (Features and Stories)
We can group tests with @Epic, @Feature, and @Stories annotations.
Here is the report result in Behaviors Section.
and that’s all. 🙂 Now, we should run the test and generate the report.
Project Code is here: https://github.com/swtestacademy/TestNGAllureReport
Step-3: Run the Test and Generate Allure Report
You can run the test with maven command. In order to do this in IntelliJ first, you should click configurations.
Select maven, and write the maven “clean test” command as shown below and then click OK.
Now, we can run the code by clicking the green run icon.
Now, it is time to generate the report!
In order to generate a report, we should install Allure command-line interpreter.
- Download the latest version as a zip archive from bintray.
- Unpack the archive to allure-commandline directory.
- Navigate to bin directory.
- Add allure to system PATH.
and finally, open a command prompt screen, go to the project directory, and write below command!
allure serve allure-results
and, you will see the beautiful Allure Test Report as shown below.
Dashboard
Categories
Suites
Graphs
Timeline
Behaviors
Packages
and we integrate Allure test framework with our automation project! 🙂
Selenium Webdriver Tutorial Series
See you in next article!
-Onur
References
[1] https://docs.qameta.io/allure/2.0/#_testng
Thank you so much Onur.
This is really helpful.
You are welcome Sudher.
Hi why not serenity-bdd
Because I am using TestNG and JAVA. If you write an article on serenity-bdd and allure integration, we will publish it to the test community.
you can check mine 😀 – https://github.com/fergadipa/serenity-allure
Hi Ferga,
Would you want to write an article on Allure and Serenity usage? Do you want to be a guest author?
Detailed information provided in the article, really good to read the article. thank you!
You are welcome! 🙂
Thanks again
Link to the serenity-bdd
http://thucydides.info/docs/serenity-staging/
Thanks. Wonderful sharing.
Thank you Asu. 🙂
Hi Onur,
Is this framework built on both Allure and ExtentReport?
Can you just use Allure but still use the annotations and tags you used in ExtentReport?
Yes, it is for both of them. But you can modify it as you wish. 🙂
Great article! Thanks for sharing. What did you do in order to have Set up and Tear down sections on your Allure report?
Hi Mircea,
You can find all codes here: https://github.com/swtestacademy/TestNGAllureReport
HI Onur, that’s a helpful article for a starter of Allure Reporting. I was also interested in using ReportPortal reports ,but now where I was able to get a good tutorials to start with, I was able to set up report portal API, but unable to get some idea how to integrate it to my existing TestNG project.
Please suggest me where I can get a step by step tutorial to start with,It would be more helpful if it by you guys.
Regards,
Amarnath
Hi Amar, at work I can not do technical work. I am doing generally managerial stuff. :/ When I will hire a test automation engineer or find a suitable time, I am planning to integrate Report Portal with TestNG and publish that work. I will keep you updated. 😉
Can I use allure offline
cause I have the same jars as you
but the @Step , @Story , …isnt working , ?
I don’t understand “offline” meaning for Allure. You can use maven and it handles everything. It automatically downloads all required libraries. You don’t need to do an extra operation for .jars. All annotations worked in my example.
Good article but it was be good if you share the code sample for listener
Can you please point to a link or article where we can generate Allure Report using Jenkins PipeLine , i.e., run my tests using Jenkins PipeLine and then generate the Allure Report using the same Jenkins Environment.
https://stackoverflow.com/questions/40379494/how-does-one-run-allure-plugin-in-jenkins-pipeline
Also at the end of your pipeline, you can try to add a batch command. You should go to your project directory and run this command “allure serve allure-results”
Great work Onur.
Once you have free time please have look at some the QA tools published by Yandex. Looks great but couldn’t get it working. If you can review that an publish a tutorial it would be really helpful for any budding automation testers googled but haven’t found anything in that area.
https://github.com/yandex-qatools/htmlelements/tree/master/htmlelements-java/src/main/java/ru/yandex/qatools
Thank you. I have also used AShot. When I will use them, I will share my experiences.
Thanks Onur for your response.
I was saying about /htmlelements a qatool used by Yandex. AShot is also a good one but not complicated and advanced as Htmlelement is my thought.
I’ll check it in detail in the future. Thank you. 🙂
could you explain in brief how to attach failure screenshot in allure using junit.
Please check here: https://github.com/allure-examples/allure-junit-example
Hi, I am not using Maven but only TestNG, is there a way I can integrate Allure reporting with TestNG ?
Thanks,
Yes without maven, you can use graddle or you can directly add the .jar files of the libraries into your project.
Thank you onur, screenshot code structure was bit tricky to embed in my structure but once embedded, it worked like charm..
You are welcome Zarrar. IF it worked, that sounds awesome.
Can you create an auto report (HTML) threw the InteliJ as in Extent Report ? that will be created in the “Target”
If you change the report directory as “Target” then I think it creates the report in Target.
I have a problem to compile my project….
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test (default-test) on project nimitz-teste: Execution default-test of goal org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test failed: There was an error in the forked process
[ERROR] org.testng.TestNGException:
[ERROR]
[ERROR] Cannot instantiate class org.testng.ITestListener
[ERROR] at org.testng.internal.ClassHelper.newInstance(ClassHelper.java:71)
[ERROR] at org.testng.internal.DefaultListenerFactory.createListener(DefaultListenerFactory.java:13)
[ERROR] at org.testng.TestRunner.initListeners(TestRunner.java:315)
[ERROR] at org.testng.TestRunner.init(TestRunner.java:274)
[ERROR] at org.testng.TestRunner.init(TestRunner.java:241)
[ERROR] at org.testng.TestRunner.(TestRunner.java:167)
[ERROR] at org.testng.SuiteRunner$DefaultTestRunnerFactory.newTestRunner(SuiteRunner.java:663)
[ERROR] at org.testng.SuiteRunner.init(SuiteRunner.java:260)
[ERROR] at org.testng.SuiteRunner.(SuiteRunner.java:198)
[ERROR] at org.testng.TestNG.createSuiteRunner(TestNG.java:1295)
[ERROR] at org.testng.TestNG.createSuiteRunners(TestNG.java:1273)
[ERROR] at org.testng.TestNG.runSuitesLocally(TestNG.java:1128)
[ERROR] at org.testng.TestNG.runSuites(TestNG.java:1049)
[ERROR] at org.testng.TestNG.run(TestNG.java:1017)
[ERROR] at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:281)
[ERROR] at org.apache.maven.surefire.testng.TestNGXmlTestSuite.execute(TestNGXmlTestSuite.java:75)
[ERROR] at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:121)
[ERROR] at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:290)
[ERROR] at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:242)
[ERROR] at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:121)
[ERROR] Caused by: java.lang.InstantiationException: org.testng.ITestListener
[ERROR] at java.lang.Class.newInstance(Class.java:427)
[ERROR] at org.testng.internal.ClassHelper.newInstance(ClassHelper.java:69)
[ERROR] … 19 more
[ERROR] Caused by: java.lang.NoSuchMethodException: org.testng.ITestListener.()
[ERROR] at java.lang.Class.getConstructor0(Class.java:3082)
[ERROR] at java.lang.Class.newInstance(Class.java:412)
[ERROR] … 20 more
[ERROR]
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
You are using surefire plugin and using forks for parallel execution. I think problem is related this. You should debug and solve this problem. Good luck.
Do you know by chance if there’s a way to configure Allure report to show only limited information on it’s main page? For example only the status circle or “Categories” pane?
I really dont know and also I could not find any info on the web as well. I am sorry. Maybe the only way is to modify allure framework library’s files.
Hi, I got report displayed correctly with above steps.
But when tried to build jenkins item, the test body is missing in the report.
Have no idea about this issue.
Could you give some advice?
I found this on the net. I hope it helps. https://stackoverflow.com/questions/40808956/how-to-configure-allure-jenkins-plugin