5W1H method to Manage your Test Automation requirements, yes but how? This article consists of two main sections these are “Describing the initial situation for your Quality Assurance processes” and “Review & Compare Testing Tools“. Let’s start to elaborate on each of them in detail.
PART 1 – Describing the initial situation for your Quality Assurance
What is expected of you? Do not forget that, Your primary goal is defining, measuring, and improving the quality of the software. Three major steps in the software testing process are planning, execution, and reporting. You are responsible for working as expected applications.
5W1H is the abbreviation summarising the following six questions: What? Who? Where? When? Why? How? This method consists of asking a systematic set of questions to collect all the data necessary to draw up a report of the existing situation with the aim of identifying the true nature of the problem and describing the context precisely. That should be asked for every project! (5W1H).
Below, I used the 5W1H method as a quality problem-resolution support tool.
Questions & Answers
- WHAT do you need?
I want to manage all test processes with a single tool with less risk and less effort. Should also support Test Automation, Manual Testing, Android & iOS Mobile Testing, Web Testing API Middleware Testing and everything must be integrated.
- WHO will use this tool?
All test team members and all other team members on projects.
- WHERE do you want to use it?
Test Automation, Remote Device, Beta Testing, UAT Testing, Manual Testing
- WHEN do you need this tool?
Getting fast visual feedback. Writing manual test cases and automating them. Finding more bugs. Having a speedy and efficient process.
- WHY do you need a Testing Tool?
Tool A is good on Manual Testing, Tool B is good on Automation, Tool C is good on Mobile Device Farm, Tool D is good but difficult to install and use, Tool E requires XScript knowledge, Tool F is the best but you can’t use tool F on your computer’s OS, Tool G is the best for Web, Tool H is free but installation & configuration will take 1 day for you and new joiners, Why you are using Tool I, just try J tool, Tool K has no test reporting abilities, infinite loop .. I don’t have enough time anymore for researching & solving the problems in the existing solution that I use. Believe me, the letters in the alphabet are not enough. There is a critical need, especially in mobile tests.
- HOW can I use it?
All popular test automation frameworks and coding languages should be supported for Automation. Mobile, Web, and API applications under testing should be supported. I want to use the tool on all Macbook / Windows / Linux OS, not limited. I want to use only Real Mobile Devices for mobile testing, not simulators/emulators. Creating codeless test scenarios should be perfect for some team members, but I want to write code when I want. I want to convert manual test cases to automated test cases when I need to. I want to do end-to-end cross-testing. I want to run parallel tests with different browsers and real devices. Easy installation and usage needed. Actually I don’t want to install anything on my computer. Built-in tool integrations (JIRA, Jenkins, Slack, Mail, Git, AWS Device Farm,..), Built-in TestData (Database, CSV, Excel,..) integrations.
Let’s dive deeper, and let’s get to know the problems that you will encounter in the future with Open Source Testing Tools.
Open-source testing tools requirements:
The most popular tools for test automation are Selenium, Appium, Cucumber, Jmeter, Postman.
1. Hardware Requirements
A powerful Macbook needed for automation engineers. Because You will need lots of things for development and debugging test automation. Mac-mini, USB extender, iOS, and Android Devices. Some guys still have to code with VBscript and use Windows laptops. It’s not your fault :(
Conclusion: There must be a different and common way of doing this, we can’t dedicate all devices to all engineers. First of all, it should be in a shareable environment.
2. Software Installations & Configurations
Example for Installation of Appium and all dependencies on your computer for mobile test automation development
- https://www.swtestacademy.com/appium-tutorial/
- https://www.swtestacademy.com/how-to-install-appium-on-mac/
Conclusion: I am not a mobile developer, do I really have to install so many things on the MacBook? I am installing more than an Android or iOS developer. How will I keep them updated? How do I upgrade to the new SDK version?
3. Get Required Access Permissions
- Test Database Access Permissions
- Backend System Access Permissions
- Git Repository Access
- Test Management Tool Access
- Issue management Access
- Notification app Access
Conclusion: If I have just started this job, I may not even know which systems I should have access to. It will also take a while for anyone new to the team to open, close, or update account/access to these systems. there must be a definitive solution to doing this in a different way.
4. Inspect Existing Test Automation Project
- *Automation Project’s Programming Language and Framework.
- Supported Platforms (iOS, Android, Web, API, Hybrid, Desktop..)
- Test Automation Scenarios
- Merge and branching strategies
Conclusion: Suppose that You feel better with Javascript automation projects. But The existing project had been written in JAVA. It will take time to learn. You can’t change the framework or language. There are thousands of lines of codes and a sprint has been started. Time is not enough for all and the team is not ready for big changes. There are common challenges in your code. False Positives & False Negatives (Flaky Tests) And your existing test automation framework doesn’t support API testing. Let’s over engine this issue. Where is the right place for your mobile automation code? Should it be in the app or outside of the app?
5. Application package to be tested
*You have to test the right packages at all times. You have to access APK(Android) and IPA(iOS Real Device) files on mobile projects.
*Package from the Latest Branch / Specified branch
Conclusion: Will I compile and create the application package to be tested, or will it come to me automatically or manually from the continuous integration tool?
6. Emulator vs Simulator vs Real Device
- Emulators and Simulators are free and they have no duration limitations like cloud services. But can’t replicate real user conditions like hardware and software configurations, work slower than real devices, consume a lot of memory and CPU on your computer.
- Emulators / Simulators are good at Unit Testing, Static Code analysis, development. They are weak at UI Regression Testing and Compatibility testing
Conclusion: Real devices provide test runs under real-world conditions. If your test is passing on an emulator/simulator that doesn’t mean it will pass on Real Device.
7. On-Premise (In-House) vs Cloud Device Farm
You need a device farm for compatibility testing and high device coverage.
Conclusion: Cloud farms have got limitations, pricing, and slowness problems. Can I believe that the mobile phone on which I ran the test is real and only dedicated to me? I prefer other users not to access my application and automation code.
8. Interactive Live Mobile App Testing on Real Device
- We are going to automate 100 percent of our regressions.
- Automation and manual test engineers are independent teams.
Conclusion: Manually exploratory testing is very important now and will remain so in the future. There is a problem of device sharing between manual and automation test engineers. Devices in on-premise device farms also have the same problem. You should not separate Manual and automation engineers. The goals of both teams are the same. Measuring the quality of the software. There is actually 1 team. You should not automate everything, Welcome to the real world.
9. Start to Automate
- Inspect ID, XPATH,.. of the mobile elements on APP, and write your automation code on IDE.
- Element locator problems (There is no ID or Name, Dynamic ID’s,..)
Conclusion: Using Appium with your favorite development IDE (IntelliJ, Pycharm, WebStorm, VisualStudio, Eclipse, Ride.. ) Can’t we do this with a single application instead of 2 different client applications with hours of installation and configurations? Should we centrally manage the Appium server and client applications? Can I get support from the development team for element locator problems? HTML concepts and basic programming knowledge are required. Reusable elements can help you.
10. Integration Requirements
- JIRA issue management / Slack notification / Email / Jenkins..
- Auto-fill and submit test run Defects to Jira
- Basic reporting with Email
- In Real-Time Get Slack notification messages for test execution.
- CI tool plugin to trigger Test Run.
- Git integration
- Cloud Device Farm integration
Conclusion: Developing dozens of integrations will require serious effort. If I cannot solve this work with open-source resources in a short time, I may have to replace the complete test automation framework. We may even need to get test consultancy.
11. Code Quality
- Use PageObject Pattern
- Fix Static Code Analysis issues.
- Apply SOLID.
- Write BDD TestCases
- Apply the GitFlow branching model
Conclusion: What is expected of me? I am not a developer.
12. TestCases in BDD (Cucumber)
- Given-When-Then formatted Acceptance criteria needed
Conclusion: JIRA issues are not in the BDD format. How can I adapt the current format to this?
13. Test Data problem
- Reading Data from Database/File/..
- Writing Data to Database/File..
- Create Test Data sets for your test automation suite
- Use Data Fakers
- Can’t generate Test Data, Mock the middleware
Conclusion: The test data problem will always be your problem. The important thing is how much support is built into which tool. Data-driven testing can help you.
14. Debugging Features
- Check your test’s screenshot, video on each step.
- Check the console logs, application logs.
- Debug your test run locally.
Conclusion: Debugging the test run is very important. Without debugging, it’s like a blind man driving a car.
15. Run Your Test
- Check the availability of the devices/browsers
- Multi-device / browser & version selection
- Check the availability of CI / CD tools
- On-demand or Scheduled test running
- Separated Test Scenarios
- Running Test Code on Docker Container
Conclusion: Where is my project built? Where is the test running on? How can I run it locally or remotely? How can I watch the stage of the test?
16. Running on Different devices (PARALLEL) / OS / Versions
- Check that all test automation results on all popular devices/versions available are as you expect.
- Running Tests with Selenium Grid
- Using Local or Remote Devices.
- Emulator / Simulator / Real Device again.
Conclusion: If all tests take two hours to run on 1 device, it should take an average of two hours to run on 100 devices. Other options are unacceptable and unexpected for me.
17. Reporting test results
- Test Case / Test Run report
- JIRA Defect Reporting
Conclusion: Reporting is not a nice to have feature, it is a must.
18. Community Support
- If you started with java, there is a lot of community support for issues/libraries. You may not have much chance for other programming languages.
- The incompatible tool versions should conflict with each other and You should get fail when you want to run your test
- You should get no support/response on some issues with open source. You are alone.
- Risk of violation and license restriction risk.
PART 2 – Review & Compare Testing Tools
We have described the initial situation for your Quality Assurance processes in the previous part. In this part, We will determine the key factors and prioritize them. We will do this by examining popular QA tools and features.
How can we find popular test tools?
Your first answer will be Google, but for reasons such as ads, Adwords, I trust Alexa more. I also got support from Alexa page ranking in this work. I had to eliminate the test tools of some successful companies whose main job was not testing. Because there were too many tools, I did not have enough time for all. I also eliminated tools that specialize only in a certain area but have no service in other areas. Because I am looking for a fully integrated solution. I am looking for a single product that contains all the solutions for me.
How can we decide the key features of the tools?
When I investigated these test tools, I found 146 different features. However, I realized that I would not use many of these features. I had determined my own needs in Part-1. I know what I need is not Artificial Intelligence. And I don’t want to change my MacBook to use a test tool. And I don’t want to go back to 2005 and develop VBscript again.
I eliminated many features that I think have no advantages (Using emulators, Xamarin support). Also, I did not add standard features such as screenshot and logging.
Item points between 0 to 5 and some for each feature. You should check the resources and tool links for item benefits.
Comparison of Main Items:
| Capabilities | Item benefits | Item Point | |
| Scripting Languages | Python | 8% Automation usage | 3(per item) |
| Java | 44% Automation usage | 5(per item) | |
| Javascript | 15% Automation usage | 4(per item) | |
| Ruby | 7% Automation usage | 2(per item) | |
| C# | 13% Automation usage | 3(per item) | |
| Others | Low Automation usage | 1 (per item) | |
| Application Under Test | iOS | Mobile Testing | 5(per item) |
| Android | Mobile Testing | 5(per item) | |
| Web | Web Testing | 5(per item) | |
| API | API / Web Services | 5(per item) | |
| Desktop | Windows Desktop App | 5(per item) | |
| Cross | All in the same project | 5(per item) | |
| Test Development Platform | on Browser (Cross-Platform) | Desktop Coverage 100% | 5 |
| on Windows OS | Desktop Coverage 77% | 3 | |
| On Win + macOS + Linux | Desktop Coverage 99% | 4 | |
| Built-in Real Mobile Device Farm | NO | NO | 0 |
| Partially with Cloud Providers | 3rd party Integration | 4 | |
| YES | Built-in + 3rd Party | 5 | |
| Programming Skills | YES |
Not required, Recommended for advanced test scripts |
5 |
| NO |
Advanced skills needed to integrate various tools |
0 | |
| Easy of installation and use | YES | Easy to setup and use | 5 |
| NO | Require installing and integrating various tools | 0 | |
| Interactive Mobile App Testing (Manual) | YES | Test remotely just with a browser (Manual Test) | 5 |
| NO | Need to purchase hundreds of physical devices (Manual Test) | 0 |
Test Tools Comparison:
Here is the comparison list.
Automation is easy with Momentum Test Tool. It was able to answer all my current needs.
I will explain “How does Momentum offer effective solutions to the problems?” in the next part.
Resources
https://applitools.com/blog/language-software-test-automation/
https://www.lambdatest.com/blog/top-7-programming-languages-for-test-automation-in-2020/
https://gs.statcounter.com/os-market-share/desktop/worldwide
Ozgur Kaya
Özgür KAYA is a Senior Software Testing and DevOps professional with 10+ years of experience. He has worked at Turkcell Global Bilgi, Veripark, Turkcell, N11, Trendyol, Emirates Airlines, and Arute Solutions, and Mobven.