“Effective Test Metrics” was one of the wonderful workshops I attended during Software Testing Conference (SOFTEC2012). The workshop was by John Fodeh and I really enjoyed it and learned a lot from it.

John talked about three main points in his workshop:

1. Measurement Fundamentals.
2. Putting Metrics into Practice.
3. People Issues.

Then, John concluded the workshop with and example and discussions which really helped a lot to understand what we learned.

1. Measurement Fundamentals:

There are several reasons behind doing measurement. We do measurement to understand, assess, predict, improve and communicate. In fact, measurement is essential to the progress of science.

Metrics:

• We need metrics to control testing activities of ISTQB Fundamental Test Process: Planning – Analysis and Design – Implementation and Execution – Evaluating Exit Criteria and Reporting – Test Closure.
• A metric should be SMART: Specific – Measurable – Acceptable – Relevant (consider the audience) – Timely (Automated metrics).
• Goal Questioned Metrics (GQM): You need to set the goals first then decide the metrics. For each goal, define the questions that must be answered to determine if you are reaching that goal.

2. Putting Metrics into Practice:

Measurement attributes can be either internal attributes or external attributes.

• Internal Attributes: Can be measured by examining the entity on its own (example: staff size – code cyclomatic complexity – number of defects found in system testing).
• External Attributes: Can be measured by examining the entity in relation to its environment (example: team productivity – application usability – inspection cost-effectiveness).

Defect Detection Percentage (DDP) is an example of a metric. It is a measure of test effectiveness and can be used in all test phases. One very important thing is that a timeframe must be determined such as 3 or 6 months after release. In addition, the value of DDP should get close to 100%. However, if it is less than 60%, then project team should work to improve.

In addition, it is also good to have a test dashboard which gives vital information in a centralized view including progress, status and any signs of trouble. An advantage of the dashboard is that it shares information across the team. some of the metrics a dashboard can contain are:

• Progress in quality.
• Test status and completion.
• Test coverage (code coverage – requirements coverage – risk coverage).
• Planned and actual resource utilization.
• Defect data (defect status – effort/time to fix – remaining defects estimation).

3. People Issues

Many software projects fail due to sociology not technology. In addition, applying metrics can be controversial especially when people are involved. This leads to reluctance to change, refusal to interact with top-down approaches and feelings of being threatened or mistreated by metrics.

Another point here is that metrics might be distorted by factors that affect:

• The environment in which the measurement is performed.
• The person/team performing the measurement.
• The measurement instrument.
• The people being measured.

One important point about metrics is that there can be misinterpretation of using metrics and that will cause side effects. This can happen due to cultural differences and a training should be provided in the organization about the metrics to avoid this. In addition, metrics should never be used to evaluate individuals performance. Instead of taking individual productivity as a metric, overall project productivity can be used. Also, use several metrics, not just one. However, you should avoid too many informational metrics.

Summary

Finally, John concluded his workshop with the following summary points:

• “Everything that can be counted does not necessarily count; everything that counts cannot necessarily be counted.” ~ Albert Einstein.
• Let your metrics be SMART.
• Metrics can help you make decisions. Not make the decisions for you.

Again, many appreciations go to John for this wonderful workshop.

All the best…