Agile Testing Question & Answers

Hi all, in this article I will answer questions about Agile Testing Mindset. These answers are based on my experience, understanding, and thoughts. I hope you will enjoy reading. Let’s start the Q&A session on Agile Testing.

Nowadays, many companies have migrated from traditional software development methodologies to Agile and Agile Testing term is using extensively in the test domain. Also, many books have been written on Agile Testing. What is Agile Testing? How do you describe it?


First of all, it is not easy to define Agile Testing explicitly in a single sentence. At least, we can define it as “Agile Testing is a software testing process that adapts to your agile software development processes”. Of course, it is necessary to open this statement. The most fundamentally changing things with agile are listed in the following order. We deploy the valuable artifacts for our customers in short periods, quality is no longer the responsibility of only one or a single group – it is the responsibility of Agile Development Team, respond and adapt more quickly and flexibly to exchanges, lighten heavy documentation and project preparation processes, increase the speed and efficiency of software development life cycle using smarter, more practical, more effective tools and solutions such as Exploratory Testing, Pairwise Testing, Desk Checking, Automation, and DevOps practices. All of these are fully compatible with Agile values: “user interactions, working software, customer collaboration and responding the change”.

As a matter of fact, Agile Testing is not a myth or legend. We continue to do the same things we used to do during the software testing process. We are trying to move away from the heavy and complicated documentation, project and test preparation processes and manual efforts which we did in traditional methods. Besides that, we are more focused on preventing bugs than finding bugs. Here, “early testing” comes into play. For “Early Test”, we can say that the test goes into the earliest phase of the software development. Test team’s feedback and reviews in Product Discovery phases, UI-UX processes, User Story creation, refinement meetings are good examples of early testing. At documentation side, we simplify or don’t use some of our test related documentation such as test scenarios, test reports, test planning, test strategy, and so on.

In agile, test periods are getting shorter. Thus, we need to create a more efficient testing process in a shorter time. Here our main aim is to release a product that is free from bugs, to minimize the product risks, to increase the confidence in the product, to increase the value and quality of the product. Here we can increase the value of the product in a variety of ways. For example; we can increase product value by performing tests such as availability, compatibility, usability, updateability, performance, security. In the agile software lifecycle, along with the speed of preparation of the requirements and the speeding up of their development, it is necessary to respond to this speed in the same way in terms of testing. Because agile teams are very close to business units, they need to understand the requirements very quickly and prioritize them. In this development life cycle, test experts should actively take part in the game with their feedbacks and actions. They should not be aware of the features developed after requirement analysis and development are done. They should know the what will be developed at the very beginning of the development lifecycle.

One of the most critical things is the team harmony, the team should act as a whole and we called this as “Whole Team Approach”. In Agile, test experts are now becoming a member of the “Agile Development Team” rather than being a member of an independent “Test Team”. We can say that the two most fundamental steps after the preparation of the User Story are development and testing. With the agile, the entire team has become responsible for all the processes of product development that have customer value. Therefore, the quality of the product is now under the responsibility of the whole team. Test experts are no longer seen as “Quality Police” or “Gate Keeper” and should not behave that way. The entire team is responsible for all tests and test types in the development lifecycle. This collaboration must be continuous and sustainable. While developers learn how to do better testing, test professionals should talk the same language with developers by technically improving themselves on system architecture, APIs, communication protocols, automation, performance, security. Of course, every member of the team has the ability to test at different levels. Test experts are the team members with the best testing competencies with their past experience, testing viewpoints, feelings, customer-focused insights and ability to force the systems to fail. For this reason, test specialists should not only test requirements at the GUI level. Already it is obvious that each member of the team is capable of doing this. Test experts should consider situations that will force the system, create alternative scenarios and find the bugs at the earliest stage. Apart from these, testers should perform usability, performance, security, stability, compatibility, etc. tests. If testers get stuck in these processes, then the whole team should help them. In this way, they will act as a whole team and they will generate a value for the customers.

Lastly, I would like to say a few things about adapting and responding to change. Within the development life cycle, requirements, features, architectural designs, tools, risks, priorities may change and all these changes must be adopted by the team. Therefore, test experts also have to be “adaptive testers”. For example, we can focus on MVP (Minimum Viable Product) in the first stages of development and accept the risks a little more, and if the product is matured, we can proceed with taking risks into account more.

Some companies do not have testers in their IT departments. Do we need test engineers in Agile?


Obviously, these are the people who have not worked with professional testers, have restricted the competencies of the testers they work with or have not helped the testers develop themselves. Because when you work with a professional tester, you get a better understanding of what the test engineer can do for you. A competent tester will provide valuable contributions to agile development teams in terms of functionality, performance, security, automation, usability, etc. from the initial stages of development to the deployment to production. In addition, some companies do not have testers but during their development process, they need tester role because they have testing needs. Because the lack of testers, developers or product owners doing testing activities instead of testers. The different roles in the development team are wearing the role of a tester and doing the testing activities. Of course, the testing which is done by developers or POs does not exactly take the place of an experienced tester’s testing. However, the people in the agile teams can be trained about testing, and gain a test perspective at a certain level. A good tester is the one who is skeptical, mitigate the risks, seeks and knows the ways to manipulate the system, thinks the corner/edge cases, tests the alternative scenarios and multiplies those alternatives. In addition, if a tester has improved himself/herself technically, this will add significant value to the team in terms of test automation, security testing, performance testing, etc. Test automation is also one of the most important parts of DevOps culture. Thus, to have testers who have good at both test knowledge and technical testing will add major strength to the agile teams.

Is cultural change necessary to be Agile? How does company culture affect to be Agile?


Absolutely! In my opinion, one the most important issues to be Agile is the company culture’s Agile adoption. First of all, I think Agile is a philosophy and this philosophy suggests that people should interact highly with each other, work in harmony as a team, avoid unnecessary procedures, be value-driven, and create practices that add value to the customers. Cultural change is a must if your company culture does not match Agile philosophy. Because the culture of the company affects its agility. Performance evaluation systems, meeting culture, people behaviors while they are listening to each other, paying attention to technical debts, giving importance to continuous improvement and sustainable excellence, all teams collaboration to each other, overtime frequencies, and so on. These kinds of behaviors in the company show us their culture and agility. I think it is very important here to start this change from the top management. It is necessary to adopt Agile culture from top to bottom. In this way, the Agile wave from top to bottom will affect the whole company and the change will be even faster. There will be a “Virtual Agile” change in a company that works with the command chain. Old performance measurement systems and tactics that cause competitions in the team against each other should never be used. Goals must be taken as a team and the team must be appreciated when it accomplishes these goals. To be a team is always having high priority. Hence, keeping team motivation at the top level will bring productivity and efficiency.

Who is an Agile Tester? What are the values and principles of an Agile Tester?


First of all, I would like to answer this question by making quotes from the Agile Testing book by Lisa Crispin and Janet Gregory. The Agile Testing book summarizes the values and principles that an Agile Tester should have. Agile tester adapts changes, establish excellent communication between technical and business teams, knows the fundamentals and essence of software testing, simplifies the processes, has an ability to analyze complex systems, mitigates the risks, find the defects in early stages of development lifecycle, enforce the quality with his/her feedback throughout the development lifecycle, adds value to product with his/her customer-focused insight, maximizes the productivity and efficiency with the technical competencies. Also, an agile tester shows right behavior in the team. This is very critical for team harmony and whole team approach.

I want to summarize the principles and values of an agile tester below. An agile tester should be:

  • Respectful
  • Professional
  • Passionate
  • Cheerful
  • Fast
  • Quick learner
  • Brave
  • Communicative
  • Explorer
  • Customer-focused
  • Happy test professional.

And also an agile tester:

  • Reduces product risks
  • Embraces and adapts changes
  • Adopts continuous improvement
  • Enforces the quality
  • Simplifies the processes
  • To be a part of the team.

Some people in the IT world say that manual testing will die in the future. Will manual testing and manual testers not exist in the future?


Actually, we should think software testing as a whole. I think testers should be competent in both manual/exploratory testing and test automation (automated-checks). A test automation engineer has to go through manual test steps before starting an automation project. Before starting to write the automation codes, it is necessary to prepare test scenarios, do the necessary work for test design, do test prioritization, create test data, setup test environments. On the contrary, manual testers should improve their technical skills. They should add several competencies to their arsenal such as API testing, Automated API testing, use the console of the browsers properly and effectively, learn some languages such as Java, C#, Python, Ruby, JavaScript, etc., learn fundamental programming skills, performance testing with several tools such as JMeter, Gatling, Locust, LoadRunner, etc., learn security and penetration testing, and DevOps practices such as CI, CD, provisioning, cloud, etc.

What kind of methods should we use for documentation in Agile?


We should lighten the documentation burden as much as possible for all kind of documentation. In Agile, the purpose of the documents is to be a guide, reminder, and helper to us when we need them. If the documents make us tired, they have started to create a significant burden instead of creating value. As a documentation tactic, when we start to prepare a document, we must first focus on the main purpose of our document. With this focus, we can start to list the main headings of our documentation. Here we can use plain text files or mind-maps. Once we have identified our main headings, we can begin to create sub-items and their details. After you identified the main skeleton of your document, it will be easier to start creating the details. We can shape our documentation in sequential periods like as Scrum sprints. We can start with creating the main headings first, then define the details of their sub-items, and then complete these details step by step. This process comprises of writing the details, finding proper images, drawing required sketches, adding related videos, quoting, referencing, etc. First, it is better to plan these to-do items and then start to work on them step by step. This methodology will be much more effective than preparing a gigantic document at once. In this way, we can also minimize the documentation pains as much as possible.

What kind of metrics should we use in Agile?


We are using several software metrics by using our toolset such as JIRA, TestRail, New-Relic, PRTG, etc. Our main indicators are; Sprint Velocities, Burn-Down Charts, Sprint Progress Bars, Sprint Boards both digital and non-digital (real board version). Apart from these, we are monitoring how many open, in-progress, completed issues do we have in development and testing. Also, we are tracking our defect statuses such as how many defects are open, resolved, closed or in-progress. Also, we are focusing their priorities, severities, root-causes, environments, re-open counts, types, etc. This tracking and monitoring help us to improve our processes and we capture some important retrospective items. We are also monitoring test statuses, activities, and test related details on TestRail Agile Test Management platform. Our test engineers are creating sprint test status reports by using these tool in agile sprints and these reports show us the status of the sprint clearly. We are taking actions according to these sprint test reports to fix the problems, eliminate the impediments, solve the bottlenecks, and help our teams. We are also using New Relic APM to monitor the performance of our applications on live environments such as critical errors, service maps statuses, response times, database and third-party integration statuses, etc. In addition to all of these, we are monitoring the alarms that our system/operations team has set up and we are checking the servers’ CPU, RAM, and other system resource utilization by PRTG. I also wrote a comprehensive software testing metrics article. You can check that article here.

What kinds of test automation strategies should we use in Agile? 


When we start to define our automation strategy, we need to know what kind of automation requirements do we have first. If your product is a word-press template, then you should focus UI and visual test automation more, if your product is a payment gateway, then it is more logical to do API test automation more. When we look at Agile Testing Quadrant or Mike Cohn’s Test Automation Pyramid, we will see that the first level of test automation should be unit testing. In this level, developers should write unit tests of their implementations. Also, they can adopt TDD (Test Driven Development) practice. It will help them to produce more defectless packages. Testers also help developers on unit tests such as reviewing their cases and providing some extra scenarios, etc. After this level, it is better to do some API level test automation. These tests run much faster than UI tests but cover less business requirements.  Then, we can do UI level test automation. We can find several Unit, API, UI test automation frameworks both commercial and open-source such as Selenium Webdriver, Capybara, Protractor, Nightwathc.js, RobotFramework, etc. If you are testing mobile apps, then you can go with Appium, XCUnit, and Espresso. HP UFT, MicroFocus Silk Test, Ranorex, Sahi Pro, TestComplete, Telerik Test Studio, etc. are some commercial solutions. There are many tools available on the market and you can select the proper one for your needs. You can do POCs and select the most appropriate solution for your environment and requirements. Also, you can register some cloud-grid services to run your web and mobile tests on the cloud such as Testinium, Saucelabs, Browserstack, TetsDroid, etc. In this way, you can leverage your Selenium Grid infrastructure and mobile test device burden easily. Also, you can do some other automation types if you really need them such as visual test automation, vulnerability/security test automation, performance test automation, etc. After writing your test automation projects, you need to integrate them with a CI development pipeline. The most know CI orchestration tools are Jenkins, TeamCity, Shippable, TravisCI, etc. In this way, you can trigger your automation projects periodically and report their results. For reporting, you can use Allure, ExtentReports,, Tesults, and similar reporting frameworks. Also, you need to write your test automation codes with best software design principles and models. In test automation, one of the most well-known patterns is Page Object Model. There are also several patterns such as Screen Play Pattern, BDD, etc. that you can use in your codes. These techniques make your automation codes more maintainable and resilient.

Which tools can we use for Agile Testing?


Oh, actually there are too many tools available in the market for several purposes. I can share some of them as follows:

  • Unit Testing: NUnit, MBUnit, JUnit, MSUnit, Xunit, etc.
  • API Automation: RestAssured, PostMan, SoapUI, Karate, etc.
  • UI Automation: Selenium, Capybara, Cucumber, RobotFramework, commercial products.
  • Visual Test Automation: ImageMagick, ApplitoolsEyes, Galen Framework, PhantomCSS, etc.
  • Performance Testing: Jmeter, Gatling, Locust etc.
  • Security and Vulnerability: ZAP, Netsparker, CAST etc.
  • APM Tools: Newrelic, AppDynamics, DynaTrace etc.
  • MOCK Service Tools: Mockito, Mockoon, Mocha, etc.
  • CI Orchestration: Jenkins, Travis CI, GO CD, Shippable, TeamCity, BitRise, FastLane, etc.
  • Containerization: Docker, Rocket (RKT)
  • Provisioning: Ansible, Puppet, Chef, etc.
  • Logging: Logstash, GrayLog, etc.
  • ELK Stack: Elastic Search, LogStash, Kibana (Search – Log – Visualization)
  • Code Quality: SonarQube, Cast, etc.
  • Deployment: AppVeyor, Octopus Deploy, shell scripting, etc.
  • Alerting: Pingdom, Nagios, PRTG (Network Monitor), etc.
  • API Gateway: Tyke, Kong, etc.

What are the success criteria to be Agile?


  • Open to change.
  • Work as a team.
  • Deliver value to the end users.
  • Optimize and simplify Software Development Life Cycle and processes.
  • Collaborate with business units.
  • To be customer-focused.
  • Adopting continuous improvement.
  • Automate routine and repetitive tasks.
  • See development and testing processes as a whole in SDLC.
  • Pay technical debt on time. Do not snooze them.
  • Establish health environment for your team.
  • Give feedback and be open to feedback.
  • To have good intentions and creating a TRUST environment.

I hope you enjoyed reading my Q&A article on Agile Testing Mindset.


Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.