In this article, we will describe and give examples on usage of IRetryAnalyzer. It reruns the Selenium TestNG tests when they are failed. If you work on test automation project, you’d know that the most difficult part of automation is the analysis of test executions. At the end of the execution, you need to analyze failed test cases and try to figure out if there’s any false positive/flaky situation caused by network glitch, time-out or some other problem.
In order to overcome this issue, we suggest you use IRetryAnalyzer interface in your TestNG based projects.
IRetryAnalyzer Class
First of all, you need to create a separate class which implements this IRetryAnalyzer like below example:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 | import org.testng.IRetryAnalyzer; import org.testng.ITestResult; /** * Created by ONUR on 17.12.2016. */ public class Retry implements IRetryAnalyzer { private int count = 0; private static int maxTry = 3; @Override public boolean retry(ITestResult iTestResult) { if (!iTestResult.isSuccess()) { //Check if test not succeed if (count < maxTry) { //Check if maxtry count is reached count++; //Increase the maxTry count by 1 iTestResult.setStatus(ITestResult.FAILURE); //Mark test as failed return true; //Tells TestNG to re-run the test } else { iTestResult.setStatus(ITestResult.FAILURE); //If maxCount reached,test marked as failed } } else { iTestResult.setStatus(ITestResult.SUCCESS); //If test passes, TestNG marks it as passed } return false; } } |
In case, you want to decrease or increase the re-run number of test cases, you need to change the maxTry value. In this example, failed test cases will run 3 times till it passes. In case it fails the third time, test execution will stop and TestNG will mark this case as failed.
Using retryAnalyzer attribute in the @Test annotation
Next step is to associate your test cases with IRetryAnalyzer. In order to do this, you need to use the method below.
@Test(retryAnalyzer = Retry.class)
public void testCase() {
}
In TestNG Listeners article I described how to use listener. I will go on with that example and Retry Class in that project. Also I changed SampleTest.java test class as follows.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 | import org.testng.Assert; import org.testng.SkipException; import org.testng.annotations.Test; import static org.testng.Assert.assertEquals; /** * Created by ONUR on 11.12.2016. */ public class SampleTest extends BaseTest{ @Test(retryAnalyzer = Retry.class) public void test1() { //Fail Scenario assertEquals(2+2,5,"Addition Problem! 2+2 must be 4!\n"); } @Test(retryAnalyzer = Retry.class) public void test2() { //Fail Scenario assertEquals(2+2,4,"Addition Problem! 2+2 must be 4!\n"); } } |
In above scenario, test1 runs 3 times and it fails during each execution. In Retry class’s retry method, when count is 3, it is smaller than maxTry and the test goes the else branch and test fails. However, test2 passed at first execution as shown below.
Using Retry Class with ITestAnnotationTransformer Interface
Due to static nature of Annotations, recompilation is needed when you want to change values. You can override this behavior at runtime with IAnnotationTransformer listener. IAnnotationTransformer is a TestNG listener which allows you to modify TestNG annotations and configure them further during runtime.
Transform method is called for every test during test run. We can use this listener for our retry analyzer as shown below:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | import org.testng.IAnnotationTransformer; import org.testng.annotations.ITestAnnotation; import java.lang.reflect.Constructor; import java.lang.reflect.Method; /** * Created by ONUR on 17.12.2016. */ public class AnnotationTransformer implements IAnnotationTransformer { @Override public void transform(ITestAnnotation annotation, Class testClass, Constructor testConstructor, Method testMethod) { annotation.setRetryAnalyzer(Retry.class); } } |
We need to add also this listener in our TestNG.xml file.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd"> <suite name="TestNG Listener Example"> <listeners> <listener class-name="Listener" /> <listener class-name="AnnotationTransformer"/> </listeners> <test name="TestNG Sample Test" preserve-order="true"> <classes> <class name="SampleTest"> </class> </classes> </test> </suite> |
From now on, we do not need to specify “@Test(retryAnalyzer = Retry.class)” this annotation. We can just only use @Test annotation without retryAnalyzer attribute.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 | import org.testng.Assert; import org.testng.SkipException; import org.testng.annotations.Test; import static org.testng.Assert.assertEquals; /** * Created by ONUR on 11.12.2016. */ public class SampleTest extends BaseTest{ //@Test(retryAnalyzer = Retry.class) @Test public void test1() { //Fail Scenario assertEquals(2+2,5,"Addition Problem! 2+2 must be 4!\n"); } //@Test(retryAnalyzer = Retry.class) @Test public void test2() { //Fail Scenario assertEquals(2+2,7,"Addition Problem! 2+2 must be 4!\n"); } } |
At this time, both test methods will be retried 3 times and at the 4th execution they all failed!
In this article, we showed you how to use Retry logic in your TestNG tests in two different ways. You can use both of the methods depends on your needs.
Test Source Code: https://github.com/swtestacademy/TestNGListener/tree/Retry
Best Regards.
Canber Akduygu & Onur Baskirt
Nice post.
Really informative
Thank you. 🙂
Finally!
Finally a tutorial with USEFUL examples, not like “System.out.println(“this is @BeforeMethod”);”
Thank you! Your site is a gem among mounts of garbage that somehow populate the top places in search.
Thank you so much Andrey. 🙂
Thanks for the Article.
Do you know how we can rerun failed Cucumber scenarios when you use TestNG. It will be really helpful. Thanks
Hi Deyan, I have not any experience on that. Did you try this code? Is it not working? I will try to check this also.
Thanks for the response Onur,
I haven’t tried the code, but I am sure it is not working because Cucumber is not using the TestNG annotations but the Cucumber ones.
Also you can not run tests in parallel with Cucumber and TestNG the same way with TestNG, you need to use this pluging:
https://github.com/temyers/cucumber-jvm-parallel-plugin/issues/31
And it looks like they are trying to develop a Retry for failed scenarios.
Hi Onur or Deyan , it will be very helpfull for me if you guys found any solution for re-running failed cucumber scenarios in testng
Please check this solution: https://stackoverflow.com/questions/21334207/rerunning-failed-cucumber-tests-using-cucumber-jvm
Can i put the retry analyzer at the class level and not for each method?
İn this example, it is in class level. You dont have any Retry annotation in Test classes. You just implement some class and add this implemented class as a listener into your testng.xml file.
Or am i missing something?
Can you please tell me at what point retry gets executed.
ie. After @Test or @After method or whenever the test gets failed.
Whenever the test gets failed.
Hi Onur,
Currently i am facing a problem in my Framework.
As while i am making a test fail Explicitly then retry is not getting executed.
Hi Onur,
Currently i am facing a problem in my Framework.
As while i am making a test fail Explicitly then retry is not getting executed.
https://github.com/Akashkansal065/FirstFrameWork
Not able to understand why the test is not getting retry.
Is there any way to execute Retry only after running of @After method for failed @Test.
Thank you for the post but its not working with @factory, just executed the @test one time and when we give counter as 3 to re-run for fail test cases, the rest get skipped.