Being in software testing for many years now, I feel that one thing that has never changed is IT managers always being in a position to justify the investment of test automation or any technical enablers to improve the testing function within their organisation. I think this is mainly due to difficulty in proving the ROI for testing.

Technically testing is the cost for a project and it doesn’t produce any tangible products, rather it is a service which improves the overall business by gaining qualitative benefits such as customer satisfaction, quicker time to market etc…

Although the benefits of test automation are clearly understood by the senior management of organisations, the common challenge is to make the right decisions on the level of test automation and right set of tools to support the test automation. One of the other common challenges is that many large organisations have completely outsourced the testing function and seem to have lost control over decisions on test automation, tool sets etc. On the other hand, the large service providers who predominantly provide testing services, are the least interested in introducing smart test automation techniques, partly worrying about the decline of head counts and also due to the overhead of training the massive resource pools in new technologies. Some of the organisations I have worked with were struggling to regain control of their testing function from their strategic partners in order to align testing with their IT transformation programmes.

Anyway coming back to the topic on how to justify the investment of test automation and service virtualization, which is a great enabler for early testing, one thing is sure: it is not going to be easy! The most fundamental thing to justify any cost is baseline metrics and processes which allow you to measure the improvements. Now you can imagine the stress involved…

I would like to share my experiences on the challenges I have faced while doing the cost-benefit analysis for investments on test automation and supporting technologies such as service virtualization. Although there are many parameters that can be chosen to analyse the cost-benefits, I think it is important to choose the ones relevant for the organisation and it is also good to consider the available baseline metrics. It is key to make the stakeholders aware of the importance of the baseline metrics in order to derive meaningful KPI(s). The following are some of the parameters that I recommend to be captured to perform cost benefit analysis.

  • The average cost of testing per project
  • The average number of testers consumed per project
  • The average test run time per project
  • The number of production defects over a set period of time
  • The average time spend by each project in the fully integrated test environments
  • The average test preparation effort per project
  • The testing coverage offered by automated testing across different levels

Even though it is easy to define what we want to measure, there are more challenges in getting them to our spreadsheets….some are listed below.

Unavailability of baseline metrics: Many times I find that bigger organisations in particular are not good at managing their key metrics. Often we walk into organisations where there are zero metrics and we need to work-around with the available information to form some baseline data. Although I get little disappointed in such situations, I completely appreciate that capturing and managing metrics is an expensive affair. What to capture, when to capture and the degree of details are all dependant on the organisation’s improvement goals. I guess the starting point is to understand the improvement goals clearly, choose the parameters to be measured, analyse the impact on the existing process and understand the cost involved to get the ball rolling.

Engaging cross functional teams: In order to continue to measure the KPIs in the defined period to prove the progress, there is a significant level of dependencies on the cross functional teams of the organisation. There will be a big hurdle on your way if the engagement of these teams is not well managed. The main thing to consider is to get the buy in from these teams by articulating the benefits to show that test automation is not an investment for testing function rather it is part of the IT transformation or IT modernization programme.

Keep a tab on intangible benefits: When some of the parameters are easy to measure and indicate the savings directly, few other intangible benefits are very tricky to measure. It is important not to ignore them. The common ones falling in this category are improvement in-build quality which is indicated by the number of defects in each stage; improvement in team morale indicated by feedback, employee retention rate; and greater customer satisfaction indicated by increased business, positive customer feedback etc..

Decisions can be made based on experience: I think we all live in the world where delay in decisions can cost huge business deals. Hence waiting for months or years to understand the complete math on ROI or cost-benefit before investing in proven technical enablers such as test automation is a risk to the business. Hence it is all about the balance between the time we spend collecting metrics, analysing cost-benefits and making timely decisions to enable technology to support critical functions of software development.

I am realising that this is a great topic where there is lot to share, it would be interesting to hear your experiences and advice.

The following two tabs change content below.
My Twitter profileMy LinkedIn profile

Priya Raju

Priya Raju is Global Head of Delivery at Sandhata Technologies. She brings a wealth of experience to Sandhata Technologies after spending over 16 years of her professional career in the IT industry. In her current role she leads a team to deliver enterprise wide change with tooling across software development and QA practices, specialising in test automation; and has developed bespoke process frameworks.