In this article, I will explain important software testing techniques which help you during test and User Acceptance Testing (UAT) processes.
Disclaimer: The knowledge in this article is based on my experiences, understanding, and observation. If you don’t agree on anything in the article, please write a comment and we will discuss the thing that you disagree.
What is User Acceptance Test (UAT)?
Product Owners (PO) and business units generally work on requirements and focus them. Thus, they can use the requirement-based and activity-based test techniques during UAT process. As we testers, we can also use these techniques in test process as well. I am going to explain some of these techniques in this article. But first, let start with what UAT testing is.
In agile methodologies, it is the test activity which performed by generally product owners after the completion of the development and software testing process of the stories. In waterfall and V-model processes, these tests are generally performed by analysts or business units. The purpose of the UAT is to determine whether the user’s requirements for the requested job have been met and to get the approval for the live deployment.
A sample entry and exit criteria for the user acceptance test are summarized in the following table.
● Code Written
● All UAT Scenarios Executed
1) User Story Testing (AGILE)
A user story can be described as a requested feature that is in the software from the perspective of the end user in agile software development life cycle. In user story, we have to specify the demand, the reason of the demand, and the user who is requesting it.
Definition of Done (DOD) defines the completion criteria such as code is done, unit test is done, testing is done, UAT is done, etc… and Scrum guide states that Scrum team (developers, testers, PO etc.) owns and responsible for DOD.
Also, acceptance criteria should be expressed clearly by POs (development team may help PO) and at least one test scenario for each acceptance criteria should be prepared in the test process of a user story and these acceptance criteria has to be tested carefully.
The test entry and exit criteria must also be defined before starting the test run. Here is an example.
|Test Entry Criteria|
● Does the requested user, the need and the reason of necessity are clearly understood?
|Test Exit Criteria |
● All Test Scenarios Executed and All Acceptance Criteria Tested
Sample User Story
User Story 1:
As a Product Owner [User], in order to advertise Kariyer.net welcome campaign [reason of demand], I would like the advertisement banner to be added to the top banner section of the Kariyer.net homepage [demand].
- Homepage speed may be reduced
- A bug in the animation of banner will affect the homepage appearance
- Continuous deletion of cookies can cause the banner to be continuously visible on the user side.
- The function of the Close Banner icon critical. It should work continuously and successfully.
- The banner loading function may be affected in the Admin panel.
Definition of Done:
- Code Writing Done
- Code Review Done
- Unit Testing Done
- UAT Done
- When Kariyer.net homepage is opened, the top banner is displayed as 200×200 for 8 seconds and then it should be seen as 60×60.
- When the user clicks on the banner, it should be directed to the Welcome page.
- If a user visited Kariyer.net more than 4 times from the same computer, AA-kobiBannerClosed cookie value should be 4 and more and banner should not be displayed.
- The top right corner of the banner must have a cross-shaped closing icon and the banner must be closed when it is clicked.
- If the banner was previously turned off by the user, it should not be displayed again.
2) Use Case Testing
A use case defines the operations that a user/actor performs in the system to achieve a specific purpose. Functional requirements of a system can be defined and managed using use cases. In this way, the scope of the desired or requested job is determined. Tests scenarios are prepared by taking into consideration the inputs and outputs of the steps determined by the user to reach a specific purpose. During the tests, the results of the tests are determined by comparing the expected outputs with the actual outputs.
When writing Use-cases, generally business language is preferred instead of technical language. Therefore they are often used in writing acceptance tests. In order to cover all requirements, at least one test scenario is prepared for each requirement. By this way, test coverage can be increased and we can measure this coverage also by using traceability matrix. In the traceability matrix, we create a matrix table with test scenarios and requirements and put a cross sign in the relevant field if it meets the requirements for each test case. The goal is to cover all the requirements.
|Scenario Name:||Changing Login Password|
|Affected Screens||Home Page, Login Page, Account Page|
|Prerequisites||An existing user on the system|
|Normal Flow:||1. The user opens the Kariyer.net homepage and clicks on the Member Login button.|
2. Login to the site with the existing username and password on the login screen.
3. Click the “My Account Settings” in the pop-up menu.
4. Click the “Change Password” link under the “Personal Information Settings:” section on the Account Settings page.
5. The old password is entered in the password change section.
6. Then, enter the new password.
7. When entering the new password, the security level on the left side is displayed as 3 levels (weak-medium-strong).
8. Enter the new password 2 times as medium-strong and click “Save”.
9. At the end of this process, the password change operation is expected to be performed successfully.
|Alternative Flows:||1. If the current password is entered incorrectly, you will be warned: “You entered your password incorrectly!”.|
2. If a weak password is entered, you will be warned: “You can not change your current password with a weak password”.
3. You can not change the password by leaving any or all of the password fields blank.
|Usage Predictions:||It is a scenario that can be used intensively by users. The functionally correct operation is critical.|
|Hypothesis:||Very complex password requirement can prevent us from being user-friendly.|
Sample Test Case
For test cases, you can use excel. For the sake of an example, all areas of the first successful test case are shown below.
Changing Password Successfully with (Moderate Complexity)
- Open Home Page.
- Click Login Button
- Go to Profile and click on “Account Settings”.
- Click “Change Password”.
- Enter the current password and new passwords (medium).
- Click Save button.
This can be done with an existing user in the system.
- User Name: firstname.lastname@example.org
- Current Password: kariyer1234+ New Password: asdf1234
It is expected that the password will be changed successfully and the information message “Changed successfully” will be informed you that the password has been changed.
3) Checklist Based Testing
In agile processes, we create a general checklist that is created independently from user stories. If there is not any risk is specified in the User Story, all or some of these checklist items may be used according to the scope of the user story. During execution of these tests, if you find a defect, you should extend the scope of the checklist by adding the failed scenario. Thus, we can increase risk items in the checklist for subsequent sprints.
- All links in the system (Web / Mobile) should work correctly.
- There should not be a grammar error in the system writings.
- Font sizes, fonts, should be as expected.
- There should not be any pictures that can not be loaded/broken in the system.
- Pictures, text, etc. The alignment between the other components must be as expected.
- All buttons must work properly and each of them directs the user to the corresponding operation.
- Each page should have the main page logo and should be redirected to the main page when clicked.
- Warning, information messages should be displayed in the correct format.
- If the page is responsive, it should be checked at all resolutions.
- All components on the site (dropdown, checkbox, radio button, etc.) should work correctly.
- Special conditions (numeric, alphanumeric, etc.) in the input fields must be checked.
- Operations can not be performed by leaving the required fields blank.
- Any operation of the site must not last more than 3 to 15 seconds.
- … etc.
4) Exploratory Testing
First of all, the exploratory testing is not a random or ad-hoc test. One of the biggest misconceptions about this test technique is that the exploratory testing is perceived as a random, non-testable, non-observable test technique. The exploratory testing is a test approach based on learning and exploring the product at the same time by using the experience, domain knowledge, analytical and intellectual knowledge of a test engineer in agile processes.
Before starting exploratory testing, a preparation should be done. Regardless of exploratory testing method selection, we should prepare a plan for the scope of functionality, tools to be used, test data, environment etc. This plan will guide the tester during test execution process. Another important point of exploratory testing is documentation is fully completed after the tests are finished.
Although it is not mandatory, “session-based testing” technique is generally preferred as an exploratory testing technique. This technique involves the following steps:
– Test Session Time (It should be a few hours)
– Session Activities
- Session setup
- Test Design and Test Execution
- Defect Searching
– The purposes of the test should be specified
– The goal of the test should be specified.
– Functionality included in the test (test report – charter) should be written.
Test Report During & After the Test Process:
– Test Report (Charter) [Specifies the test function.]
– The Person who Performed Test
– Start Date and Time
– Session Metrics (Metrics Collected During Test Implementation and Defect Searching)
– Test Data
– Test Notes
- http://www.satisfice.com/tools/htsm.pdf (English)
- https://www.linkedin.com/pulse/kesfederek-test-yapmak-alper-mermer (Turkish)
5) Experienced Based Testing
This testing technique based on the knowledge, skills, and experience of the person who will make the test. In this testing technique; test planning, test strategy, test inputs, and test scenarios are determined by the experience of the person performing the test. In order to prefer this technique, it must be an experienced candidate with sufficient technical and business knowledge to perform this test.
It is easier to understand what is going right or wrong during the test because the experiences gained in past projects are taken into consideration. This person can perform tests by using techniques like exploratory testing, which will make it easier to use past experience and intellectual/analytical knowledge. When we have very short test execution time or there is a lack of sufficient documentation on the project, etc. make sense to use this test technique. If the system being tested comprises of high risks, it is not preferable to use the Experienced Based Testing technique alone because the context of the requirement should be totally covered.
6) User Journey Test
The User Journey test is taking into account the various road maps and journeys of a typical user on the system. In these tests, the most critical journeys a user would make within the site are determined and then the scenarios of these journeys are written. Thus, the user’s interactions with the system are covered as much as possible. These tests are usually “end-to-end” tests, so they may take more time to run than other tests, but the coverage percentage of these tests are higher than the other ones.
Since the “User Journey” tests are comprehensive and extensive tests, the numbers are relatively low compared to other tests. Especially, “user journey” tests can be created taking into account the most critical “use-case” scenarios. The most plausible behavior here is that positive basic scenarios, called “happy-path”, should be run first.
For an example on kariyer.net, it is an important “user-journey” test for a user to enter the site, login, then search for the “sales representative” keyword and apply to the first job ad successfully. In this “user journey”, opening the site, logging in, making a successful call in the search bar, opening a related announcement page and then submitting the application from this announcement page is covered. As you can see here, the broad coverage of the “user journey” tests beneficial in early detection of critical faults in the software development process especially before starting extensive testing. Unlike User Story tests, user journey tests are not tied to user stories. When there are changes introduced by a new user-story, existing user journey tests are updated taking these new changes into account. New user stories rarely cause new user journey tests. For this to happen, new features must be added to the system.
7) Risk-Based Testing
One of the most fundamental objectives of risk-based test techniques is to find the most critical and most important errors as early as possible with lowest cost. Risks are the things that we do not know exactly what will happen, but we know the probabilities of what might happen. Briefly, they are possible problems. When these possibilities are unknown, they are called uncertainties. Therefore, we can think of the general definition of the magnitude of the risk is the multiplication of the likelihood of problems and their impact. Thus, in risk-based tests, we prioritise and test the most error-prone functionalities.
Magnitude of Risk = Likelihood * Impact
The early adoption of the risk-based test approach in projects is important for the early detection of critical problems.
The most basic steps of the risk-based test are summarized below:
1- First, risks are identified and a prioritized risk list is prepared.
2- Make a test plan according to the prioritized risk list and tests are executed for each risk.
3- As a result of the tests, some risks eliminate and some of them arise. New risks are tested by considering their test effort.
At this point, our most fundamental goal will be to find the most important defects.
If you are responsible for testing a product with a very high failure cost, you will need to do a very detailed risk analysis. Statistical models can be used here. One of the most known model is the Failure Mode Effect Analysis (FMEA) model.
In short, if we will be referring to the FMEA model, we calculate the risk score by taking 3 metrics with 5 scales. These:
|Data loss, hardware or security issues||Urgent||1|
|Loss of functionality, not even have a workaround||High||2|
|Loss of functionality, which has a workaround||Medium||3|
|Partial loss of functionality||Low||4|
|Cosmetic or trivial||N/A||5|
|Total loss of system value||Urgent||1|
|Unacceptable system value loss||High||2|
|Possible reduction in system value||Medium||3|
|Acceptable reduction in system value||Low||4|
|Negligible reduction in system value||N/A||5|
|Negligible reduction in system value||Urgent||1|
|Some users are affected||High||2|
|Possible influences for some users||Medium||3|
|Limited impact of few users||Low||4|
|Impact not to be deducted from actual use||N/A||5|
All these three qualities (Severity, Priority, and Probability) are calculated separately within themselves, then these values are multiplied together to obtain a Risk Priority Number (RPN).
Risk Priority Number (RPN) = S * P * L
Based on this RPN value, we can determine the scope of the test. Lower RPN indicates higher risk.
8) Heuristic Risk-Based Testing by James Bach
(First published in Software Testing and Quality Engineering Magazine, 11/99 Copyright 1999, James Bach)
You can find the details in this link: http://www.satisfice.com/articles/hrbt.pdf
I just want to give a summary of Heuristic Risk-Based Testing.
At the beginning of your projects, your risk analysis may be incomplete or incorrect in some degree because it is not possible to estimate everything 100% at first. However, as your project progresses and your product improves, your estimates and risk analysis will become increasingly stronger. According to James Bach, the two most critical factors for risk are experiences and teamwork. Over a period of time, products or technologies begin to reveal characteristic problems. It is important to observe and learn about them. Besides, it is very critical to do risk analysis with people with different perspectives. Also, using risk-based lists helps us to negotiate with the management about test workforce effectiveness. We can show them that we use existing test workforce for the most critical points of the product or we may need to explain them we need the extra workforce to cover all the risk areas. These risk-based lists give you extra power to negotiate with the management.
In this article, we covered very important test techniques. I hope you enjoyed to read it. All the things in this article based on my understanding and experiences. If you disagree anything, please do not hesitate to write a comment and If you convince me I will fix the thing that you are right. 😉
You can also read my Q&A on Agile Testing Mindset article here: http://www.swtestacademy.com/agile-testing/