Hi all, for a long time I wanted to create a new section on swtestacademy.com and share the knowledge, experience, and advice of software testing gurus to all of you and this is the first episode of these interview series which is called as “Test Masters”.

For the first episode, I contacted with Alan Richardson because I stepped into the world of technical testing and test automation with his courses. After those courses, I always followed him and take him as a role model. Thus, I wanted to start this episode series with Alan.

alan richardsonMaybe, some of you don’t know Alan so I want to give a little bit information about him. First, Alan is an “Evil Tester” and he helps people test better. Alan has dedicated his entire professional career to improving the software testing process. He has a great deal of expertise in Agile and Automated Testing, as well as manual exploratory, technical testing, and performance testing. He has spoken at many international conferences and has been involved in testing for almost 20 years. Alan is the author of Selenium Simplified, Java for Testers, Dear Evil Tester, and Automating and Testing a REST API books. He also offers online training for Selenium 2 WebDriver Basics with Java and Technical Web Testing 101. You can find Alan’s courses, books, and websites here.

Onur: Hi Alan, welcome to the first episode of Test Masters. You called yourself as an Evil Tester. What does an Evil Tester do? Why should we be an Evil Tester?


Thanks Onur.

Evil Tester started as a cartoon character that I used to draw short one or two panel cartoons that allowed me to vent my frustration when on a variety of projects. Because testing often wasn’t taken seriously, or we weren’t given enough time to test, or testing was almost a separate project run in parallel to a Software Development project with minimum collaboration. Testing was viewed as a “Necessary Evil” rather than a value part of the Software Development process.

In order to make a change on the projects I was on I had to behave differently, and communicate differently from other people. I wanted to talk in common sense terms, challenge the traditional concepts that were taken for granted and find more efficient ways of working. That often seemed like a radical approach and wasn’t always viewed as ‘good’. This was slightly before Lean, Agile and Exploratory really took off as terms in our industry so I had to find a concept that worked for me to build those ways of thinking into my process. A non-mainstream set of attitudes is often classified as “Evil” by the current establishment.

I really use the term to describe an attitude that frees people to approach their work differently. Rather than adopting the mainstream definitions and rules without question. Think them through to their core. Describe them in your words, rather than the words of authority. Be prepared to do whatever it takes to test the software. Be prepared to do the things that other people won’t do, that other people can’t even imagine doing. Be prepared to ask the questions that others will not ask.

Regarding Software Development, I conceptualized the terms Good and Evil as more of a Yin and Yang balance. Very often we label the balancing attribute with negative terms. Software Testing and Software Testers very often must be prepared to offer a balance.

I don’t think this immediately works for everyone. It takes time to develop skills to help you view things differently, to avoid a dualistic view point of ‘right and wrong’ and instead view things as systems. It takes time to learn how to communicate challenges without annoying the listener or isolating yourself from the project. I do try and share the books and resources I’ve used over the years to learn these skills on my blogs and in my videos and training.

Ultimately, I hope that everyone finds their own way of adding balance to the projects they work in, and learns to communicate what they do and how they learn to do it, so that we can all learn from each other. Evil Testing isn’t a useful concept for everyone. But it worked for me.

Onur: What are the most critical things in software testing?


That is a hard question. As a tester I think I value most my ability to build models systems and my ability to question those models. But I think that underpinning that is a continued desire to learn and improve.

Software Testing requires a diverse range of skills and techniques, and I find that I am continuing to add to those, so my studying never stops. Recently I’ve been studying a lot of Marketing and have found the world of Strategic Planning to offer a lot of parallels with the position of Software Testing.

If I am interviewing testers for a role then I look for understanding of general software testing concepts and understanding them well enough that they can explaining them in their own words rather than parroting definitions. I look for practical ability to test – demonstrated by sitting down at a computer and testing software. I look for the ability to explain what and why they are doing – rather than applying techniques by rote, they use the context of the application and goals of the exercise to trigger different thought processes and approaches. I also look for the attribute of constant improvement so that I can see they are continually reflecting and learning from what they do.

Onur: How should we model our test and automation processes in Agile development world? Which strategies should we use?


I don’t think Agile has really changed my model of Testing or Automating. I think, because I was deconstructing traditional models prior to working in Agile that I was able to adapt to Agile more readily. To answer the question, I probably have to go slightly meta.

When we test, we try to validate that certain desired conditions have been met, and we try to explore for any unexpected conditions, or new conditions, so we can decide if they are desired or not. We do this on Agile projects by validating acceptance criteria and exploring stories for emergent or unexpected behavior and for any Risks we have identified. We also did this on non-Agile projects, but very often the approach used emphasized agreeing up front what people would look for, therefore concentrating more on acceptance criteria validation, and de-emphasizing the exploratory and experimentation aspects of our work.

Automating hasn’t changed as much as I would have liked. People still talk about ‘automating tests’ rather than ‘automating processes’. When we automate processes we tend to build more flexible abstraction layers around which we can add the ability to assert on conditions to use the automated execution to support testing. When we automate tests, we often build frameworks which emphasize communication of ‘test results’ and abstractions are often rigid in support of the reporting, rather than flexible in terms of the automating.

What has changed is that for Agile we want to move away from a lot of upfront analysis and documentation, and instead learn how to build and communicate models incrementally, document our testing as we test rather than in up front plans, and completion reports. As a project, we need to learn how to automate the conditions that are important to us on an ongoing basis as we build and test the product, rather than as a separate exercise in parallel to the main development work.

Therefore, I recommend people read more about Systems Thinking and Cybernetics because these topics cover adaptability and flexibility which are traits we adopt when working in Agile.

Onur: Do you believe in software testing metrics and KPIs? Do they work? Should we use them?


I think that depends in how you interpret those words. Traditionally a Software Testing Metric might be

  • Number of Test Cases written
  • Number of Test Scripts written
  • Number of Test Scripts Executed per day
  • etc.

And if you are working on a long project then you need to track work done to know if you are likely to meet the deadlines that you have set for yourself.

Converting work into numbers, so you can estimate progress and work/time left is important on long projects, otherwise it is hard to make decisions about how to change.

Another approach is to work in smaller chunks so you don’t have to measure as much because you can adopt a more qualitative and more subjective approach to decision making.

Unfortunately, I think people very often use Metrics and KPIs as a proxy for actually managing people, work and projects so I tend to try to create a system such that we can use different ways of measuring progress than the traditional numbers. I often think that a lot of the emphasis on metrics was simply related to the tools used. The tools were built on an Entity Relationship basis, and so counting numbers of ‘things’ and ‘number of relationships added’ was easy. And then we valued the numbers, rather than the activity of Testing that the Entities and Relationships were supposed to support.

If you know how to manage well then you can use whatever metrics you need to give you a quantitative view of the project to help your decision making. Because you will also choose to measure things such that the process of measurement does not detrimentally impact your project. And you will change what you measure to meet your goals and give you the information you need to make effective decisions.

If you do not know how to manage well then you will use whatever numbers the tools kick out, or you will look online and find ‘metrics’ that ‘other people’ seem to use. And you will ask for those numbers to put into spreadsheets so that you can communicate numbers to people who don’t understand what you are doing, rather than actually communicating the process of testing.

I personally use a lot of numbers in my work. I track my testing with timestamps, make a lot of notes, track the time I spend on different areas of work. I track progress against goals so I can adjust my progress. When I manage teams I try to collate information on progress automatically. But I’d rather build ways of working that make that unnecessary.

In theory Agile is a system that makes this easier. We have Stories. Stories have some estimates and Acceptance Criteria. We split the Story into the tasks we think we have to do finish the story. We can see what tasks are done, and what tasks are not done. We can see if our estimates match reality. This is done on very fine grained time levels – hours, days. So it is easier to adjust our approach.

Within Agile, because the tools we use often model Entities like: Stories, Tasks, Defects, etc. Some people will convert those numbers into metrics. But those metrics probably don’t help as much as qualitatively assessing the progress of the work by the people doing the work.

Onur: How do software managers motivate testers?


I’ve lost at least one job interview with this answer. They don’t. A manager’s job is to remove the impediments that demotivate. This might mean: working practices, lack of training, lack of tooling, unrealistic expectations, poor communication, etc.

Some of the testers may be motivated by external factors, in which case the manager can try to make those available to help the testers. This might include appropriate feedback and communication. It will vary from tester to tester. It is also possible to view this as removal of an impediment that demotivates; some people are demotivated when they do not receive positive feedback on their work, some people are demotivated when they don’t know the plans for the company, etc.

Motivation is a highly individual construct, a manager needs to be careful when they try to motivate their team:

  • I have personally found spot bonuses to be quite demotivating, because the size of the bonus was often related to ‘position’ (job role) rather than work or impact
  • I personally am not motivated by cakes, lunch, drinking sessions after work, team games, bonding sessions, competitions, gifts etc. Some people are.

Find out what improves your individual testers work conditions and help them do the best job they can.

Onur: How should testers motivate themselves?


First, recognize what factors impact your motivation. Then take action to make them happen.

I am more motivated when I see things being finished. Therefore I track ‘successes’ and ‘failures’ so that I have completion on aspects of my work. And then I can reflect on what made them a success or failure.

I am motivated to learn new knowledge which I can apply. Therefore I like to have to opportunity to experiment, work with new technology, try different approaches, read new books, etc. Therefore I make my managers aware of what is working for me, and what is not. They can then remove those impediments to help me maintain my motivation.

If you ever find yourself in a situation where you are demotivated. Take action. Identify what is impacting your ability to generate motivation, if you can’t change the things that impact you. Ask for help – either from team mates, or management.

Onur: Should testers learn all technical testing topics such as Automation, Performance, Security, CI/CD, Cloud, Provisioning, Virtualization etc. or Should they focus and deep dive to one or some of them?


I leave that entirely up to each person to decide.

Some people are not interested in those topics. If they choose not to learn them then that will change the options they have later on in their career. They may have no choice but to enter management and earn more money than their technical peers. They may then have no choice but to rise up the hierarchy of leadership and buy a fast car or a big house.

Some people are interested, and can see how they can apply those technical topics in their work to improve their approach. They then prioritize what they will learn based on how quickly they can apply the knowledge and benefit from it. This might help them work on projects that are technically challenging and pay more than their non-technical or management peers.

I have gained benefits from learning some of those topics. Some I have deep dived, some I have surface knowledge. Some I don’t know. I still have  a to do list where some of those topics are on it.

Each person takes responsibility for their own learning. Recognizing that each thing we have chosen to learn, or not learn, is a choice. Recognizing that each learning is prioritizing that action above something else. We choose what we will sacrifice, and what we will gain. We each must accept responsibility for our own development.

Onur: What are your most important advice for testers?


Take responsibility for your approach to testing. Never stop learning. Share what you learn. Pay attention to the trends in testing. Investigate and implement trends, don’t trust them blindly, trust your experiments. Communicate testing in your own words. Apply your unique interests and personality to your testing. Find out who you are. Be yourself.

Onur: Alan, it was a matchless privilege to host you at Test Masters series. Thank you so much.

Alan Richardson’s Courses, Books, and, Websites