Estimating Test Effort
Posted: July 13, 2010 Filed under: Planning, Software Metrics, Tips and Tricks | Tags: Testing Leave a comment »One of the best known techniques are the ones provided by Caper Jones and David Longstreet. According to Caper Jones,
Total number of test cases = (Count of Function Points) raised to the power of 1.2.
David Longstreet provides a similar formula for computing the number of UAT test cases:
Total number of UAT test cases, which is = 1.2 x (Count of Function Points).
As with every such empirical model, one has to use them as a guide and evaluate them with real data. In my own experiences with web based (Microsoft dot net platform), the numbers are slightly different, and approximate better as follows:
Total Number of test cases = (Function Point Count) raised to the power 1.05
Total number of UAT test cases = 1.35 x (Function Point Count)
Once the count of the test cases has been found, apply the productivity factors for test case authoring and test case execution to arrive at the total testing effort.
The following is the data for a 3000 function point project on which the above formula have been tweaked.
Unit Test Cases : 13,000
Integration Test Cases : 5,000
System Test Cases : 13,000
UAT: : 4,100
Total : 35,100
References: Estimating Test Cases and Defects by David Longstreet
Software Estimation Rules of Thumb by Caper Jones (pdf)
Software Metrics- IV
Posted: June 22, 2010 Filed under: Software Metrics, Uncategorized | Tags: Project Turnaround 2 Comments »While I will write a set of posts on my experience in turning around projects, I would like to illustrate here how metrics can be used to analyze the depth of the problem and also provide a very quick plan for stabilizing the project very quickly.
A simple defect analysis based on the cause of the defects can shed light on what are the key areas that need to be addressed.
In the following diagram from an actual web- based application development project, for example, it is very clear that the process for regression testing needs to be improved as defects because of its inefficiency are 22%. It is also clear that the substantial numbers of defects (23%) are on account of requirements (missed + ambiguous) and another 17% of the defects pertain to insufficient test cases. Addressing these three areas would have an immediate impact on the quality of testing. Given cost and schedule constraints, it may not be required to address all the causes and instead concentrate only on these three factors that address 72% of the defects.
Software Metrics-III: An unorthodox metric
Posted: June 9, 2010 Filed under: Software Metrics | Tags: Agile, Methodology, Software, Waterfall Leave a comment »The feedback received on the project’s artifacts was related not only to the defects in the work products but a major part was non-defect comments (see diagram below). We were able to demonstrate to the customer through an analysis that all review comments were not defects. A detailed classification was also provided to substantiate the data. This enabled the customer’s project manager to go back to his team and guide them in controlling the non-defect review comments. This resulted in reduction of re-work by the team and enhanced the customer’s perception about the quality of the deliverables.
Key Steps:
- Consolidate all review comments in a single document (a tool like MS Excel can also be used).
- Categorize each review comment by the phase, artifact and type.
- Prepare a visual representation (example pie chart) of review comments and include as part of the weekly status
Outcome: Help the customer analyze the project feedback by categorizing and visually representing the review comments for a better perspective. This will help both the team and the customer by reducing re-work and enhancing perceptions about deliverables.
Software Metrics-II
Posted: June 4, 2010 Filed under: Software Metrics Leave a comment »The figure below offers a good starting point to determine what the most important metrics from a testing standpoint could be. The cost of detecting and fixing a defect escalates as the defect is detected later in the project. The obvious implication is that testing should aim to find defects earlier in the lifecycle. Another is that defects should be predictable- this can often be achieved based on historical data and a few heuristics.
The x axis in the graph below shows the phase of the project, and the y axis shows the relative cost of finding a defect in the software.
Source: Roger Pressman, Software Engineering, ed 5
The next post will identify some of the metrics that can be derived from the above diagram. Meanwhile, can you take a shot at identifying what kind of testing metrics can be useful in reducing the cost of defects?
Software Metrics- I
Posted: June 2, 2010 Filed under: Software Metrics | Tags: Software Metrics Leave a comment »When you can measure what you are speaking about, and can express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind: It may be the beginning of knowledge, but you have scarcely in your thoughts advanced to the stage of science. -Lord Kelvin
Software metrics used to measure attributes associated with a software application or a project. The purpose of measuring is self evident- one measures to improve. In a software development project, one measures attributes that help to improve quality, speed up execution and reduce costs, in other words, they should be useful to make software development faster, better and cheaper. This series of posts discusses some of the software metrics that are particularly important from a testing perspective and the benefits and challenges associated with gathering, analyzing and using these metrics.
What is to be measured?
“The risk with any metric is that people will come to see it as a description of reality, rather than a tool for a conversation about reality… one metric or another can function well only when managers know why they are measuring and for whom… In the world of social value-creation, context is king.” (The Economist survey of wealth and philanthropy from the February 25th, 2006 issue). Source
While the talk of software metrics abounds- indeed a number of companies do have a metrics plan- few implement it, or implement it in a systematic form. One reason could be that significant costs are associated with such measurements, one estimates puts the cost as 4-8% of the total development budget1. Another is that while metrics are useful primarily from a management perspective, the source of the data are the technical people on the project. More often than not, the latter are so much in the thick of meeting deadlines that data is either not collected sufficiently or not on time, when it is too late to collect it. Metrics, particularly related to defects, are certainly collected when the applications or product does not exhibit stability or cause too many issues for the end user. However, more often than not this is as part of a damage control action, rather than as part of a systematic practice of gathering data. On the other end of the spectrum, data that might be gathered as part of practice may not relate or even obfuscate the intended objective. It is, therefore, important that while data gathering and metrics analysis is done as part of the routine, care is taken to gather only a few but key metrics.
1. NE Fenton and SL Pfleeger, Software Metrics: A Rigorous and Practical Approach, 2ndedition, Boston: PWS Publishing, page 28
