Agile Testing Question & Answers

Hi all, I will answer questions about the Agile Testing Mindset in this article. These answers are based on my experience, understanding, and thoughts. I hope you will enjoy reading it. Let’s start the Q&A session on Agile Testing.

Many companies have migrated from traditional software development methodologies to Agile, and the Agile Testing term is used extensively in the test domain. Also, many books have been written on Agile Testing. What is Agile Testing? How do you describe it?

Onur:

First, it is not easy to define Agile Testing explicitly in a single sentence. At least, we can define it as “Agile Testing is a software testing process that adapts to your agile software development processes.” Of course, it is necessary to open this statement. The most fundamentally changing things with agile are listed in the following order. We deploy the valuable artifacts for our customers in short periods, quality is no longer the responsibility of only one or a single group – it is the responsibility of the Agile Development Team to respond and adapt more quickly and flexibly to exchanges, lighten heavy documentation and project preparation processes, increase the speed and efficiency of software development life cycle using smarter, more practical, more effective tools and solutions such as Exploratory Testing, Pairwise Testing, Desk Checking, Automation, and DevOps practices. These are fully compatible with Agile values: “user interactions, working software, customer collaboration and responding to the change.”

Agile Testing is not a myth or legend. We continue to do the same things we used to do during the software testing process. We are trying to move away from the heavy and complicated documentation, project and test preparation processes, and manual efforts we did in traditional methods. Besides that, we are more focused on preventing bugs than finding bugs. Here, “early testing” comes into play. For “Early Test”, we can say that the test goes into the earliest phase of the software development. Test team’s feedback and reviews in Product Discovery phases, UI-UX processes, User Story creation, and refinement meetings are good examples of early testing. On the documentation side, we simplify or don’t use some of our test-related documentation, such as test scenarios, test reports, test planning, and strategy.

In agile, test periods are getting shorter. Thus, we need to create a more efficient testing process in a shorter time. Here our main aim is to release a product that is free from bugs, minimize the product risks, increase confidence in the product, and increase the value and quality of the product. Here we can increase the product’s value in various ways. For example, we can increase product value by performing tests such as availability, compatibility, usability, upgradeability, performance, and security. In the agile software lifecycle, along with the speed of preparation of the requirements and the speeding up of their development, it is necessary to respond to this speed in the same way in terms of testing. Because agile teams are very close to business units, they need to understand the requirements and prioritize them quickly. Test experts should actively participate in the game with their feedback and actions in this development life cycle. They should not be aware of the features developed after requirement analysis and development are done. They should know what will be developed at the beginning of the development lifecycle.

One of the most critical things is team harmony, the team should act as a whole, and we call this the “Whole Team Approach.” In Agile, test experts are now becoming a member of the “Agile Development Team” rather than an independent “Test Team.” We can say that the two most fundamental steps after preparing the User Story are development and testing. With agile, the entire team has become responsible for all the processes of product development that have customer value. Therefore, the quality of the product is now under the whole team’s responsibility. Test experts are no longer seen as “Quality Police” or “Gate Keeper” and should not behave that way. The entire team is responsible for all tests and test types in the development lifecycle. This collaboration must be continuous and sustainable. While developers learn to do better testing, test professionals should speak the same language as developers by technically improving themselves on system architecture, APIs, communication protocols, automation, performance, and security. Of course, every member of the team can test at different levels. Test experts are the team members with the best testing competencies with their past experience, testing viewpoints, feelings, customer-focused insights, and ability to force the systems to fail.

For this reason, test specialists should not only test requirements at the GUI level. It is obvious that each team member is capable of doing this. Test experts should consider situations that will force the system, create alternative scenarios and find the bugs at the earliest stage. Apart from these, testers should perform usability, performance, security, stability, compatibility, etc. tests. If testers get stuck in these processes, then the whole team should help them. In this way, they will act as a whole team and generate value for the customers.

Lastly, I would like to say a few things about adapting and responding to change. Within the development life cycle, requirements, features, architectural designs, tools, risks, and priorities may change, and all these changes must be adopted by the team. Therefore, test experts also have to be “adaptive testers.” For example, we can focus on MVP (Minimum Viable Product) in the first stages of development and accept the risks a little more, and if the product is matured, we can proceed with taking risks into account more.

Some companies do not have testers in their IT departments. Do we need test engineers in Agile?

Onur:

These people have not worked with professional testers, have restricted the competencies of the testers they work with, or have not helped the testers develop themselves. Because when you work with a professional tester, you better understand what the test engineer can do for you. A competent tester will provide valuable contributions to agile development teams in terms of functionality, performance, security, automation, usability, etc., from the initial stages of development to the deployment to production. In addition, some companies do not have testers, but they need a tester role during their development process because they have testing needs. Because of the lack of testers, developers or product owners do testing activities instead of testers. The different roles in the development team are wearing the role of a tester and doing the testing activities. Of course, the testing done by developers or POs does not exactly take the place of an experienced tester’s testing. However, the people in the agile teams can be trained about testing and gain a test perspective at a certain level. A good tester is the skeptical one who mitigates the risks, seeks and knows how to manipulate the system, thinks about the corner/edge cases, tests the alternative scenarios, and multiplies those alternatives. In addition, if a tester has improved himself/herself technically, this will add significant value to the team in terms of test automation, security testing, performance testing, etc. Test automation is also one of the most important parts of DevOps culture. Thus, testers who are good at both test knowledge and technical testing will add major strength to the agile teams.

Is cultural change necessary to be Agile? How does company culture affect to be Agile?

Onur:

Absolutely! One of the most important issues to be Agile is the company culture’s Agile adoption. First, I think Agile is a philosophy that suggests that people should interact highly with each other, work in harmony as a team, avoid unnecessary procedures, be value-driven, and create practices that add value to the customers. Cultural change is necessary if your company culture does not match Agile philosophy. Because the culture of the company affects its agility. Performance evaluation systems, meeting culture, people behaviors while listening to each other, paying attention to technical debts, importance to continuous improvement and sustainable excellence, all teams collaboration with each other, overtime frequencies, and so on. These kinds of behaviors in the company show us their culture and agility. I think starting this change from the top management is very important here. It is necessary to adopt Agile culture from top to bottom. This way, the Agile wave from top to bottom will affect the whole company, and the change will be even faster. There will be a “Virtual Agile” change in a company that works with the command chain. Old performance measurement systems and tactics that cause competition in the team against each other should never be used. Goals must be taken as a team, and the team must be appreciated when it accomplishes these goals. To be a team is always having high priority. Hence, keeping team motivation at the top level will bring productivity and efficiency.

Who is an Agile Tester? What are the values and principles of an Agile Tester?

Onur:

First of all, I would like to answer this question by making quotes from the Agile Testing book by Lisa Crispin and Janet Gregory. The Agile Testing book summarizes the values and principles that an Agile Tester should have. Agile tester adapts to changes, establish excellent communication between technical and business teams, knows the fundamentals and essence of software testing, simplifies the processes, can analyze complex systems, mitigates the risks, find the defects in the early stages of the development lifecycle, enforce the quality with his/her feedback throughout the development lifecycle, adds value to the product with his/her customer-focused insight, maximizes the productivity and efficiency with the technical competencies. Also, an agile tester shows the right behavior in the team. This is very critical for team harmony and the whole team approach.

I want to summarize the principles and values of an agile tester below. An agile tester should be:

• Respectful
• Professional
• Passionate
• Cheerful
• Fast
• Quick learner
• Brave
• Communicative
• Explorer
• Customer-focused
• Happy test professional

And also an agile tester:

• Reduces product risks
• Embraces and adapts to changes
• Adopts continuous improvement
• Enforces the quality
• Simplifies the processes
• To be a part of the team.

Some people in the IT world say that manual testing will die in the future. Will manual testing and manual testers not exist in the future?

Onur:

We should think of software testing as a whole. I think testers should be competent in both manual/exploratory testing and test automation (automated checks). A test automation engineer has to go through manual test steps before starting an automation project. Before starting to write the automation codes, it is necessary to prepare test scenarios, do the necessary work for test design, prioritize, create test data, and set up test environments. On the contrary, manual testers should improve their technical skills. They should add several competencies to their arsenal, such as API testing, Automated API testing, using the console of the browsers properly and effectively, learning some languages such as Java, C#, Python, Ruby, JavaScript, etc., learning fundamental programming skills, performance testing with several tools such as JMeter, Gatling, Locust, LoadRunner, etc., learn security and penetration testing, and DevOps practices such as CI, CD, provisioning, cloud, etc.

What kind of methods should we use for documentation in Agile?

Onur:

We should lighten the documentation burden as much as possible for all kinds of documentation. In Agile, the purpose of the documents is to be a guide, reminder, and helper to us when we need them. If the documents make us tired, they have started to create a significant burden instead of creating value. As a documentation tactic, when we start to prepare a document, we must first focus on the main purpose of our document. With this focus, we can start to list the main headings of our documentation. Here we can use plain text files or mind-maps. Once we have identified our main headings, we can begin to create sub-items and their details. After you identify your document’s main skeleton, it will be easier to start creating the details. We can shape our documentation in sequential periods like Scrum sprints. We can start with creating the main headings first, then define the details of their sub-items, and then complete them step by step. This process comprises of writing the details, finding proper images, drawing required sketches, adding related videos, quoting, referencing, etc. First, it is better to plan these to-do items and then start to work on them step by step. This methodology will be much more effective than preparing a gigantic document at once. In this way, we can also minimize the documentation pains as much as possible.

What kind of metrics should we use in Agile?

Onur:

We use our toolset’s several software metrics, such as JIRA, TestRail, New-Relic, PRTG, etc. Our main indicators are; Sprint Velocities, Burn-Down Charts, Sprint Progress Bars, and Sprint Boards, both digital and non-digital (real board version). Apart from these, we are monitoring how many open, in-progress, completed issues we have in development and testing. Also, we are tracking our defect statuses, such as how many defects are open, resolved, closed, or in progress. Also, we are focusing on their priorities, severities, root causes, environments, re-open counts, types, etc. This tracking and monitoring help us improve our processes and capture some important retrospective items. We also monitor test statuses, activities, and details on the TestRail Agile Test Management platform. Our test engineers are creating sprint test status reports by using these tools in agile sprints, and these reports clearly show us the sprint’s status. We are taking action according to these sprint test reports to fix the problems, eliminate the impediments, solve the bottlenecks, and help our teams. We are also using New Relic APM to monitor the performance of our applications in live environments, such as critical errors, service map statuses, response times, database and third-party integration statuses, etc. In addition to all of these, we are monitoring the alarms that our system/operations team has set up and checking the servers’ CPU, RAM, and other system resource utilization by PRTG. I also wrote a comprehensive software testing metrics article. You can check that article here. http://www.swtestacademy.com/software-testing-metrics-kpi/

What kinds of test automation strategies should we use in Agile? 

Onur:

When we define our automation strategy, we need to know what kind of automation requirements we have first. If your product is a word-press template, then you should focus on UI and visual test automation more; if your product is a payment gateway, then it is more logical to do API test automation. When we look at Agile Testing Quadrant or Mike Cohn’s Test Automation Pyramid, we will see that the first level of test automation should be unit testing. At this level, developers should write unit tests of their implementations. Also, they can adopt TDD (Test Driven Development) practice. It will help them to produce more defectless packages. Testers also help developers with unit tests, such as reviewing their cases, providing some extra scenarios, etc. After this level, it is better to do some API-level test automation. These tests run much faster than UI tests but cover fewer business requirements.

Then, we can do UI-level test automation. We can find several Unit, API, and UI test automation frameworks, both commercial and open-source such as Selenium Webdriver, Capybara, Protractor, Nightwathc.js, RobotFramework, etc. If you are testing mobile apps, you can go with Appium, XCUnit, and Espresso. HP UFT, MicroFocus Silk Test, Ranorex, Sahi Pro, TestComplete, Telerik Test Studio, etc. are some commercial solutions. Many tools are available on the market, and you can select the proper one for your needs. You can do POCs and select the most appropriate solution for your environment and requirements. Also, you can register some cloud-grid services to run your web and mobile tests on the cloud, such as Testinium, Saucelabs, Browserstack, TetsDroid, etc. This way, you can easily leverage your Selenium Grid infrastructure and mobile test device burden.

Also, you can do some other automation types if you need them, such as visual test automation, vulnerability/security test automation, performance test automation, etc. After writing your test automation projects, you must integrate them with a CI development pipeline. The most know CI orchestration tools are Jenkins, TeamCity, Shippable, TravisCI, etc. You can trigger your automation projects periodically and report their results. For reporting, you can use Allure, ExtentReports, ReportPortal.io, Tesults, and similar reporting frameworks. Also, you need to write your test automation codes with the best software design principles and models. One of the most well-known patterns in test automation is Page Object Model. There are also several patterns such as Screen Play Pattern, BDD, etc. that you can use in your codes. These techniques make your automation codes more maintainable and resilient.

Which tools can we use for Agile Testing?

Onur:

Oh, too many tools are available in the market for several purposes. I can share some of them as follows:

• Unit Testing: NUnit, MBUnit, JUnit, MSUnit, Xunit, etc.
• API Automation: RestAssured, PostMan, SoapUI, Karate, SuperTest, PactumJS, etc.
• UI Automation: Selenium, Playwright, Cypress, Cucumber, RobotFramework, commercial products.
• Visual Test Automation: ImageMagick, ApplitoolsEyes, Galen Framework, PhantomCSS, etc.
• Performance Testing: Jmeter, Gatling, Locust, K6, etc.
• Security and Vulnerability: ZAP, Netsparker, CAST, etc.
• APM Tools: Newrelic, AppDynamics, DynaTrace, etc.
• MOCK Service Tools: Mockito, Mockoon, Mocha, Nock, etc.
• CI Orchestration: Jenkins, Travis CI, GO CD, Shippable, TeamCity, BitRise, FastLane, etc.
• Containerization: Docker, Rocket (RKT)
• Provisioning: Ansible, Puppet, Chef, etc.
• Logging: Logstash, GrayLog, etc.
• ELK Stack: Elastic Search, LogStash, Kibana (Search – Log – Visualization)
• Code Quality: SonarQube, Cast, etc.
• Deployment: AppVeyor, Octopus Deploy, shell scripting, etc.
• Alerting: Pingdom, Nagios, PRTG (Network Monitor), etc.
• API Gateway: Tyke, Kong, etc.

What are the success criteria to be Agile?

Onur:

• Open to change.
• Work as a team.
• Deliver value to the end users.
• Optimize and simplify Software Development Life Cycle and processes.
• Collaborate with business units.
• To be customer-focused.
• Adopting continuous improvement.
• Automate routine and repetitive tasks.
• See development and testing processes as a whole in SDLC.
• Pay technical debt on time. Do not snooze them.
• Establish a healthy environment for your team.
• Give feedback and be open to feedback.
• To have good intentions and create a trusting environment.

I hope you enjoyed reading my Q&A article on Agile Testing Mindset.

Thanks.
-Onur