Software Quality Report 2018 is finally ready! As you may all noticed, we conducted a questionnaire on QA activities at the end of 2018. Thanks to all responders, we got more than 90 people from all over the world (Brazil, India, US, Turkey, Netherlands,etc…).
Let’s see the results…
How do you describe your company?
Here’s the overview of the participants. Nearly half of the participant is working in IT Services companies. Then financial services come as second.
What is Your Title in Your Company?
We are on the right track, most of the participants are working in QA area.
Which automation activities do you have in the company?
As we can see, almost everybody has Web automation projects in their companies. Then it’s followed by API Automation by 70%. We can conclude that companies believe the value that both activities bring into the projects. Half of the people who participated stated that they are doing Mobile Automation. That’s a very good number as the mobile world is getting bigger and bigger, we need to put additional value in mobile testing. It’s likely that it will be increased next year. One thing that’s surprising for me is that Unit Testing is also into play. But the most surprising on is 1 over 3 people are doing desktop application automation. We think that Appium added value in desktop application automation.
Do you do Performance Testing Activities? Which tools are you using?
Firstly, 25% of the people stated that they don’t do performance testing. What we recommend is that you should start immediately :) It’s an expected result, JMeter is the leading tool on performance test projects. It’s a great tool with more than 20 protocols supported. JMeter is followed by 2 commercial and 1 free tool. Commercial tools are Visual Studio and HP Loadrunner. The free one is Gatling. We believe Gatling will surpass those two commercial tools in the near future as it’s becoming increasingly popular. It’s also good to see Locust on that list. People want to adopt new tools. This is a tool we also like to write about.
Here are some articles on
JMeter – https://www.swtestacademy.com/category/performance-testing/jmeter/
Gatling.io – https://www.swtestacademy.com/category/performance-testing/gatling/
Which development language are you using in your automation projects?
As expected, Java is the leading development language on test automation project. It’s followed by C# and Javascript. There are many libraries built on JS, we expect it to be adopted more and surpass C# in near future. We also can conclude that companies don’t stick to one development language, there are companies using more than 2-3 different language for the different type of automation. That’s a very powerful approach. There’s also a very big community who does test automation with python. We forget to add it in the answers but they wrote us the name. We can say learn Python!
Which automation tools/libraries are you using?
It’s no surprise that Selenium is the leading library/tool or whatever you want to call it. Almost every participant use Selenium WebDriver. We don’t believe there will be a decrease in its popularity in a few years as for next-generation Selenium IDE is also on the way. Then it’s followed by Appium which is the defacto solution for mobile automation. Then we have others. Cucumber is the leading BDD tool when it comes to automation. We also have Specflow coming from a few people but Cucumber surpasses it. For some time, we hear people adopting WebDriver.io and it’s on the list with a big percentage. 20% of the participant uses it. Then we have the others like HP, Robotframework, Sikuli, and Cypress.
We expect this list to be a little bit stable for a few years :)
Do you use Simulators or Real Devices for mobile test automation?
%32 of companies says they don’t do mobile automation. That’s also a known fact from a previous question. When we take a look at the other response. The number of participants who use simulators is not that low. We expect this number to decrease next years as Cloud companies find a cheap and easy solution for people to use real devices on the cloud.
For compatibility issues, which browsers are tested in web automation projects?
People need Chrome. That’s for sure after seeing the answer to this question. Then people test their web application with Firefox also. But it’s 52%. We think that Chrome will be the leading browser for a few years. Internet Explorer is tested by 33 % of the participant even it’s a little bit painful :) Good to see little participant are also testing against Opera browser too :)
Do you use any cloud providers for web or mobile automation?
After seeing the answer to this question, we believe that cloud solution provider companies have a lot to conquer. People still try to build their own device farm or browser grids to execute their tests. BrowserStack is the most adopted tool followed by Cross-browser Testing and Saucelabs. Maybe we did not reach some countries that those tools are adopted mostly but it seems those companies have many customers waiting for them. There we also see Testinium which is a Turkish company providing its tool with test automation services.
We got a response from people who use Zalenium (https://github.com/zalando/zalenium) and Selenoid (https://github.com/aerokube/selenoid).
You can read about Selenoid in those articles also
https://www.swtestacademy.com/selenoid-tutorial/
https://www.swtestacademy.com/docker-selenium-tutorial/
Do you have any CI in your company?
This is the question we think we know the answer. That’s right. Jenkins is the leading one followed by GitLab (26%). That’s a very big difference. Jenkins is a very mature tool with a big ecosystem. Not easy to leave it when you love it :) But we see that people are using more than 2 CI tool at the same time.
We can say that mostly everybody adopted a CI tool.
We also have many Jenkins article in that link. You might want to check them out.
We plan to do another survey next year. Stay tuned!
Canberk Akduygu
Canberk Akduygu is a Test Lead working in the Netherlands