From Continuous Testing to Continuous Learning – Deriving Values from KPIs

The accelerated development of software has become a business essential; development teams are clambering to keep up with the pace at which business demands new changes. DORA’s Accelerate State of DevOps 2019 report states one of the findings that software development efficiency heavily drives the organization’s performance. 

Delivering software quickly, reliably, and safely is at the heart of technology transformation and organizational performance. We see continued evidence that software speed, stability, and availability contribute to organizational performance (including profitability, productivity, and customer satisfaction). Our highest performers are twice as likely to meet or exceed their organizational performance goals. [1]

The industry has gone through many SDLC transformations over the past decade – the classic waterfall cycles evolved to Agile methodology, further leading towards faster development lifecycles like DevOps, Continuous Delivery, and its newest avatar Continuous Testing. Continuous Testing brings real-time methods to measure the software development process and map it to the “big picture” of the business objectives.

Organizations today need a cultural transformation in testing the applications – a paradigm shift that requires tracking beyond the “pass-fail ratio” to recognizing the potential risks associated with it. The maturity and adaptation of the development methodology largely depend on an organization’s technical milieu, and it always comes at a cost. This article attempts to add a perspective to align the business and quality KPIs in Continuous Testing, taking the economic and business aspects of a software product into account and enable the teams to make informed decisions based on metrics.

“It is not worth doing if it is not worth measuring!”

The old saying in the business world has never been more relevant. Continuous Testing advocates the idea of a feedback loop – experiment, learn, act. It is a system that relies extensively on understanding customer viewpoints and sharing feedback with application developers and business stakeholders. It improves the observability of an application and enables data-driven decision-making. There are several metrics extensively used by the application development teams for measuring the quality. However, the principal aspect of choosing the preferred set of KPIs largely depends on the context of quality. The interpretation of quality for an end-user and an application developer is substantially different. Teams choose different sets of KPIs based on their definition. However, it is of paramount importance to the account that the metrics should lead to the three aspects of decision-making.

• Business value: Comprehensive benchmarks are those where one can capture their business values and prove that targets are getting met – like customer adaptation, engagement, revenue, etc.

• Product quality: Software has evolved from being an enabler for business processes to giving a competitive edge, which leads to the point where the quality of modern software is of utmost significance.

• Product Delivery: Today’s modern organizations demand faster time to market to remain competitive. The essence of Continuous Testing is to bring down the lead time of higher-quality releases while reducing production defects rate and shortening the time to market.

Connecting the dots…

The difference in perspectives of business stakeholders and development teams poses a challenge to know where the team stands; hence it is equally difficult to measure improvements. The business needs are split into Projects, Epics, Features, and lastly to the Stories. At this last level, developers work, metrics are available. A well-defined set of metrics can aggregate and lead to the key performance indicators which complete the circle. Hence, the identification of metrics plays a significant role in this process.

continuous-testing-kpis

Building the Story…

Here is an indicative set of metrics that can aggregate to the three pillars of Continuous Testing measures. Broadly, there are three sources of information.

1. Numbers captured from Agile tools (Jira, Rally, etc.): Some of the key numbers available from these tools are Story Point Estimations, Business Value, Priority, etc.

2. Statistics available in Continuous Integration and Deployment Tools: These tools are probably the most valuable source of essential metrics for quality KPIs. The Continuous Integration pipelines can provide the measures like – automated test case count at various test phases, code coverage, static code analysis report, vulnerability assessment report, cyclometric complexity, performance test reports, build quality, and many others. Deployment tools provide information like Application Downtime, Deployment Duration, etc.

3. Vitals captured through Production Monitoring Tools: These provide essential information like Application Health, Application Outage, increase/decrease in application traffic, Product Incidents, etc.

Towards Continuous Learning…

All these stats are available in real-time. The metrics collected can lead to the driving aspects of decision-making.

metrics-insights

Let us discuss some of the performance indicators, which can be of interest.

• Time to market – We can derive this by combining a few metrics collected through the agile tools (tracking the feature level breakdowns) with a few others available from CI/CD tools (pull requests, deployment status, etc.).

• Cost of feature delivery – Continuous Testing metrics can effectively lead to the cost of product delivery. The total cost of product delivery is the sum of development efforts (can be derived through metrics from Agile tools), with infrastructure cost (resource usage is available from monitoring tools). It also includes the cost of maintenance, which we can also derive from the metrics captured.

• Conversion Rate Optimization – Business stakeholders can leverage A/B Testing to experiment with new features, which can help in optimizing a website or webpage to improve customer experience.

• Test Coverage – Continuous Testing provides a different outlook of software testing. The idea here is to look beyond functional and non-functional distinctions to the overall quality of the product. Code coverage, cyclometric complexity, automated test cases, performance tests, security test metrics combined with production incidents can provide test coverage. This KPI is vital in risk management and optimization of the test process.

• Test Process Compliance and best practices – The quality metrics can be aggregated at the product/system level to comprehend the testing process, compliance, and adherence to software testing best practices. It can also highlight the adaptability and variations in the development efficiency by introducing new capabilities in the testing process. It is essential in test process optimization.

metrics-dashboard

Experiment, learn, act!

A poor quality software product has a direct impact on business revenue, as well as brand image. But quality comes at a cost.

The right set of KPIs empowers businesses in data-driven decision-making. When we validate the testing process, it is hard to determine which processes are more critical to others. An analytical risk-based testing strategy can help in optimizing quality costs. Risk (like quality) is also subjective and highly dependent on the context of the product, application (even can be specific to the projects!). Risk analysis factors in likelihood and impact in determining risk level. The goal is to minimize the level of risk under a tolerance threshold. There are several factors like reliability, maintainability, security, compliance that can contribute to risk assessment. Continuous Testing with advanced analytics exposes application risks early. The risk assessment can lead to a bunch of risk mitigation tasks. This process also highlights the value added by each part of the quality process.

So, the knowledge gained assists in optimizing the testing process. The test reports support risk assessment. The feedback loop helps in understanding redundancy, process problems, instability root cause, and mitigate those. The findings are accurate and actionable. It leads to design efficient tests, introduce new testing processes, optimize test suites with increased reliability. The visibility and agility help in experimenting with test process optimization and act fast on the outcome.

That’s all for today!

A common challenge among the delivery team and the product team is their opposing thought processes. There is a lack of transparency as well as clarity on the bigger picture. The KPIs answer many questions- the insights help drive better decisions by giving visibility while respecting the perspectives of different stakeholders. I hope this blog post helps you and your team get another viewpoint of the quality KPIs, and understand the power of metrics.

Happy Testing!
Abhi Nandan

Bibliography

[1] [Online]. Available: https://services.google.com/fh/files/misc/state-of-devops-2019.pdf.

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.