In this article, I will explain important software testing techniques which help you during test and User Acceptance Testing (UAT) processes.
Disclaimer: The knowledge in this article is based on my experiences, understanding, and observation. If you don’t agree with anything in the article, please write a comment, and we will discuss the thing you disagree with.
What is User Acceptance Test (UAT)?
Product Owners (PO) and business units generally work on requirements and focus on them. Thus, they can use the requirement-based and activity-based test techniques during the UAT process. As testers, we can also use these techniques in the test process. I am going to explain some of these techniques in this article. But first, let’s start with what UAT testing is.
In agile methodologies, it is the test activity performed by product owners generally after the completion of the stories’ development and software testing process. In waterfall and V-model processes, these tests are generally performed by analysts or business units. The purpose of the UAT is to determine whether the user’s requirements for the requested job have been met and to get approval for the live deployment.
A sample entry and exit criteria for the user acceptance test are summarized in the following table.
|
Entry Criteria ● Code Written |
Exit Criteria ● All UAT Scenarios Executed |
1) User Story Testing (AGILE)
A user story can be described as a requested feature in the software from the end user’s perspective in the agile software development life cycle. In the user story, we have to specify the demand, the reason, and the user requesting it.
Definition of Done (DOD) defines the completion criteria such as code is done, a unit test is done, testing is done, UAT is done, etc… and Scrum guide states that the Scrum team (developers, testers, PO, etc.) owns and responsible for DOD.
Also, acceptance criteria should be expressed clearly by POs (the development team may help PO), and at least one test scenario for each acceptance criteria should be prepared in the test process of a user story. These acceptance criteria must be tested carefully.
Before starting the test run, the test entry and exit criteria must also be defined. Here is an example.
|
Test Entry Criteria ● Does the requested user, the need, and the reason of necessity are clearly understood? |
Test Exit Criteria ● All Test Scenarios Executed and All Acceptance Criteria Tested |
Sample User Story
User Story 1:
As a Product Owner [User], to advertise Kariyer.net welcome campaign [reason of request], I would like the advertisement banner to be added to the top banner section of the Kariyer.net homepage [request].
Risks:
- Homepage speed may be reduced
- A bug in the animation of the banner will affect the homepage appearance
- Continuous deletion of cookies can cause the banner to be continuously visible on the user side.
- The function of the Close Banner icon is critical. It should work continuously and successfully.
Impact Analysis:
- The banner loading function may be affected in the Admin panel.
Definition of Done:
- Code Writing Done
- Code Review Done
- Unit Testing Done
- UAT Done
Acceptance Criteria:
- When Kariyer.net homepage is opened, the top banner is displayed as 200×200 for 8 seconds, and then it should be seen as 60×60.
- When the user clicks on the banner, it should be directed to the Welcome page.
- If a user visited Kariyer.net more than four times from the same computer, the AA-kobiBannerClosed cookie value should be four and more, and the banner should not be displayed.
- The top right corner of the banner must have a cross-shaped closing icon, and the banner must be closed when clicked.
- If the user previously turned off the banner, it should not be displayed again.
2) Use Case Testing
A use case defines a user/actor’s operations in the system to achieve a specific purpose. Functional requirements of a system can be defined and managed using use cases. This way, the scope of the desired or requested job is determined. Test scenarios are prepared by considering the inputs and outputs of the steps determined by the user to reach a specific purpose. During the tests, the results of the tests are determined by comparing the expected outputs with the actual outputs.
When writing Use-cases, generally, business language is preferred instead of technical language. Therefore they are often used in writing acceptance tests. At least one test scenario is prepared for each requirement to cover all requirements. In this way, test coverage can be increased, and we can also measure this coverage using a traceability matrix. In the traceability matrix, we create a matrix table with test scenarios and requirements and put a cross sign in the relevant field if it meets the requirements for each test case. The goal is to cover all the requirements.
| Scenario No | S.1 |
| Scenario Name: | Changing Login Password |
| Connected Requirements | R.4 |
| Affected Screens | Home Page, Login Page, Account Page |
| Prerequisites | An existing user on the system |
| Normal Flow: |
1. The user opens the Kariyer.net homepage and clicks on the Member Login button. 2. log in to the site with the existing username and password on the login screen. 3. Click the “My Account Settings” in the pop-up menu. 4. Click the “Change Password” link under the “Personal Information Settings:” section on the Account Settings page. 5. The old password is entered in the password change section. 6. Then, enter the new password. 7. When entering the new password, the security level on the left side is displayed as three levels (weak-medium-strong). 8. Enter the new password twice as medium-strong and click “Save.” 9. At the end of this process, the password change operation is expected to be performed successfully. |
| Alternative Flows: |
1. If the current password is entered incorrectly, you will be warned: “You entered your password incorrectly!”. 2. If a weak password is entered, you will be warned: “You can not change your current password with a weak password.” 3. You can not change the password by blanking any or all password fields. |
| Usage Predictions: | It is a scenario that can be used intensively by users. The functionally correct operation is critical. |
| Hypothesis: | Very complex password requirements can prevent us from being user-friendly. |
| Notes: |
Sample Test Case
For test cases, you can use excel. For the sake of an example, all areas of the first successful test case are shown below.
Test Name
Changing Password Successfully with (Moderate Complexity)
Test Steps
- Open Home Page.
- Click Login Button
- Go to Profile and click on “Account Settings.”
- Click “Change Password.”
- Enter the current password and new passwords (medium).
- Click the Save button.
Prerequisites
This can be done with an existing user in the system.
Test Data
- User Name: [email protected]
- Current Password: kariyer1234+ New Password: asdf1234
Test Priority
High
Expected Results
It is expected that the password will be changed successfully, and the information message “Changed successfully” will be informed you that the password has been changed.
3) Checklist-Based Testing
In agile processes, we create a general checklist independently from user stories. If there is no risk specified in the User Story, all or some of these checklist items may be used according to the scope of the user story. During the execution of these tests, if you find a defect, you should extend the scope of the checklist by adding the failed scenario. Thus, we can increase risk items in the checklist for subsequent sprints.
Example
- All links in the system (Web / Mobile) should work correctly.
- There should not be a grammar error in the system writings.
- Font sizes, and fonts, should be as expected.
- There should not be any pictures that can not be loaded/broken in the system.
- Pictures, text, etc. The alignment between the other components must be as expected.
- All buttons must work properly, and each directs the user to the corresponding operation.
- Each page should have the main page logo and be redirected to the main page when clicked.
- Warning, information messages should be displayed in the correct format.
- If the page is responsive, it should be checked at all resolutions.
- All components on the site (dropdown, checkbox, radio button, etc.) should work correctly.
- The input fields’ special conditions (numeric, alphanumeric, etc.) must be checked.
- Operations can not be performed by leaving the required fields blank.
- Any site operation must not last more than 3 to 15 seconds.
- … etc.
4) Exploratory Testing
First of all, exploratory testing is not a random or ad-hoc test. One of the biggest misconceptions about this test technique is that exploratory testing is perceived as a random, non-testable, non-observable test technique. Exploratory testing is a test approach based on learning and exploring the product simultaneously using a test engineer’s experience, domain knowledge, and analytical and intellectual knowledge in agile processes.
Before starting exploratory testing, preparation should be done. Regardless of exploratory testing method selection, we should prepare a plan for the scope of functionality, tools to be used, test data, environment, etc. This plan will guide the tester during the test execution process. Another important point of exploratory testing is documentation is fully completed after the tests are finished.
Although it is not mandatory, the “session-based testing” technique is generally preferred as an exploratory testing technique. This technique involves the following steps:
Main Activities:
– Test Session Time (It should be a few hours)
– Session Activities
- Session setup
- Test Design and Test Execution
- Defect Searching
- Reporting
– The purposes of the test should be specified
– The goal of the test should be specified.
– Functionality included in the test (test report – charter) should be written.
Test Report During & After the Test Process:
– Test Report (Charter) [Specifies the test function.]
– The Person who Performed the Test
– Start Date and Time
– Session Metrics (Metrics Collected During Test Implementation and Defect Searching)
– Test Data
– Test Notes
– Results
– Errors
Recommended resources
- http://www.satisfice.com/tools/htsm.pdf (English)
- https://www.linkedin.com/pulse/kesfederek-test-yapmak-alper-mermer (Turkish)
5) Experienced-Based Testing
This testing technique is based on the knowledge, skills, and experience of the person who will do the test. In this testing technique, test planning, strategy, test inputs, and test scenarios are determined by the experience of the person performing the test. To prefer this technique, an experienced candidate with sufficient technical and business knowledge must perform this test.
It is easier to understand what is going right or wrong during the test because the experiences gained in past projects are considered. This person can perform tests using techniques like exploratory testing, making it easier to use past experience and intellectual/analytical knowledge. When we have a very short test execution time or lack sufficient documentation on the project, etc. makes sense to use this test technique. If the system being tested comprises high risks, it is not preferable to use the Experienced Based Testing technique alone because the context of the requirement should be covered.
6) User Journey Test
The User Journey test considers the various road maps and journeys of a typical user on the system. In these tests, the most critical journeys a user would make within the site are determined, and then the scenarios of these journeys are written. Thus, the user’s interactions with the system are covered as much as possible. These tests are usually “end-to-end” tests, so they may take more time to run than other tests, but the coverage percentage of these tests is higher than the others.
Since the “User Journey” tests are comprehensive and extensive, the numbers are relatively low compared to other tests. Especially, “user journey” tests can be created considering the most critical “use-case” scenarios. The most plausible behavior here is that positive basic scenarios, called “happy path”, should be run first.
For example, on kariyer.net, it is an important “user-journey” test for a user to enter the site, log in, search for the “sales representative” keyword and apply to the first job ad successfully. In this “user journey,” opening the site, logging in, making a successful call in the search bar, opening a related announcement page, and then submitting the application from this announcement page is covered. As you can see here, the broad coverage of the “user journey” tests is beneficial in the early detection of critical faults in the software development process, especially before starting extensive testing. Unlike User Story tests, user journey tests are not tied to user stories. When a new user story introduces changes, existing user journey tests are updated, taking these new changes into account. New user stories rarely cause new user journey tests. For this to happen, new features must be added to the system.
7) Risk-Based Testing
One of the most fundamental objectives of risk-based test techniques is to find the most critical and most important errors as early as possible with the lowest cost. Risks are things that we do not know exactly what will happen, but we know the probabilities of what might happen. Briefly, they are possible problems. When these possibilities are unknown, they are called uncertainties. Therefore, we can think of the general definition of the magnitude of the risk as the multiplication of the likelihood of problems and their impact. Thus, we prioritize and test the most error-prone functionalities in risk-based tests.
The magnitude of Risk = Likelihood * Impact
The early adoption of the risk-based test approach in projects is important for the early detection of critical problems.
The most basic steps of the risk-based test are summarized below:
1- First, risks are identified, and a prioritized risk list is prepared.
2- Make a test plan according to the prioritized risk list, and tests are executed for each risk.
3- As a result of the tests, some risks are eliminated, and some arise. New risks are tested by considering their test effort.
At this point, our most fundamental goal will be to find the most important defects.
You will need to do a detailed risk analysis if you are responsible for testing a product with a very high failure cost. Statistical models can be used here. One of the most known models is the Failure Mode Effect Analysis (FMEA).
In short, if we refer to the FMEA model, we calculate the risk score by taking three metrics with five scales. These:
Severity
| Description | ImportanceLevel | Point |
| Data loss, hardware, or security issues | Urgent | 1 |
| Loss of functionality, not even have a workaround | High | 2 |
| Loss of functionality, which has a workaround | Medium | 3 |
| Partial loss of functionality | Low | 4 |
| Cosmetic or trivial | N/A | 5 |
Priority
| Description | Importance Level |
Point |
| Total loss of business value | Urgent | 1 |
| Unacceptable system business loss | High | 2 |
| Possible reduction in the business value | Medium | 3 |
| Acceptable reduction in the business value | Low | 4 |
| A negligible reduction in the business value | N/A | 5 |
Likelihood
| Description | Importance Level |
Point |
| A negligible reduction in the system value | Urgent | 1 |
| Some users are affected | High | 2 |
| Possible influences for some users | Medium | 3 |
| Limited impact of few users | Low | 4 |
| Impact not to be deducted from actual use | N/A | 5 |
All these three qualities (Severity, Priority, and Probability) are calculated separately within themselves, then these values are multiplied together to obtain a Risk Priority Number (RPN).
Risk Priority Number (RPN) = S * P * L
Based on this RPN value, we can determine the scope of the test. Lower RPN indicates higher risk.
8) Heuristic Risk-Based Testing by James Bach
(First published in Software Testing and Quality Engineering Magazine, 11/99 Copyright 1999, James Bach)
You can find the details at this link: http://www.satisfice.com/articles/hrbt.pdf
I just want to give a summary of Heuristic Risk-Based Testing.
At the beginning of your projects, your risk analysis may be incomplete or incorrect because it is impossible to estimate everything 100% at first. However, as your project progresses and your product improves, your estimates and risk analysis will strengthen. According to James Bach, the two most critical factors for risk are experiences and teamwork. Over a period of time, products or technologies begin to reveal characteristic problems. It is important to observe and learn about them. Besides, it is very critical to do risk analysis with people with different perspectives. Also, using risk-based lists helps us to negotiate with the management about test workforce effectiveness. We can show them that we use the existing test workforce for the most critical points of the product, or we may need to explain to them we need the extra workforce to cover all the risk areas. These risk-based lists give you extra power to negotiate with the management.
In this article, we covered very important test techniques. I hope you enjoyed reading it. All the things in this article are based on my understanding and experiences. If you disagree with anything, please do not hesitate to write a comment, and If you convince me, I will fix the thing that you are right about. ;)
You can also read my Q&A on Agile Testing Mindset article here:
http://www.swtestacademy.com/agile-testing/
Thanks.
-Onur
Onur Baskirt is a Software Engineering Leader with international experience in world-class companies. Now, he is a Software Engineering Lead at Emirates Airlines in Dubai.
Nice Explanation!!
Thank you @Bharat.