5

Most defect density metrics are like this:

Number of bugs / Size (KLOC, Story points, Function points..)

I have read that it measures the effectiveness of QA, but I do not get it:

  • I can have a super senior developer and a sloppy QA. Therefore, the sloppy QA guy finds nothing because there is nothing.
  • I can have a bad developer and a good QA guy who finds most of the issues.

Therefore, how could this be measuring the effectiveness of QA? In my view, it reflects both the quality of dev and effectiveness of QA.

Improve this question

4 Answers 4

Reset to default
7

I have read that it measures the effectiveness of QA but I do not get it

It is surely a good idea to take any such statement about metrics with a grain of salt.

The only way I can think of to use the given metrics to measure the effectiveness of QA, is to take the same piece of code, give it to different QA people and let them independently test it. In theory, the more bugs they find, the better the QA (however, in reality the severity of the bugs found should also be incorporated).

Of course, by doing so the size of the code is fixed, so using the density as a number is quite pointless, you could actually just use "Number of bugs found by QA" as a metrics.

As you correctly have noted by yourself, as soon as you vary the QA persons and the subject under test, the metric does not tell you anything reliable about the effectiveness of the QA any more.

Since you did not give any reference where you found this questionable statement, I did a quick web search to see what others wrote about this metrics. It brought me to this page, mentioning two uses for the defect density metrics:

  • For comparing the relative number of defects in various software components so that high-risk components can be identified and resources focused towards them.

  • For comparing software/products so that quality of each software/product can be quantified and resources focused towards those with low quality.

"Effectiveness of QA" is nowhere mentioned. So according to this source, defect density is a metrics for quantifying quality aspects of the software, not of the development or QA process.

(And if you read something different in a text book, you better ask the authors of the book what they had in mind with their statements.)

Improve this answer
7
  • Well but in reality, you have no time to for testing a piece of code by separate QAs. What I mean is, if the Defect density is low - I have either good developers or good QA. (considering density only of those found after release). So how is this QA effectiveness? Commented Oct 8, 2018 at 14:52
  • @JohnV: my answer is not about the time you have in reality, it is about the stated metrics and the correct interpretation of what you read about it - from a theorectical point of view. If "in reality, you have no time to for testing a piece of code by separate QAs", then you probably should not try to use this metrics for measuring effectiveness of QA, it won't give you useful results. Commented Oct 8, 2018 at 14:59
  • Yes and that is my point - nobody has time for this in practice. And yet this metric is in various books mentioned as "measuring effectiveness of QA". If the Defect density is 0 (meaning no bugs reported months after release), either my QA is amazing or my developers are godlike :) Commented Oct 8, 2018 at 15:06
  • @JohnV: well, don't believe anything just because it is written in some textbook ;-) Commented Oct 8, 2018 at 15:12
  • So I assume I will carry on with using this metric to indicate our general trend with regards to defects in our system, instead of using it to assess my QA. Commented Oct 8, 2018 at 15:14
3

It was probably refering to defects found after release. ie by users not QA.

In this case, it is a measure of QA effectiveness as poorly coded work would be easier to find bugs in and hence not release. Very well, but not perfect code, might have a couple of minor, hard to find bugs slip through, but not many.

Of course it could also measure project manager-careing-more-about-deadlines-than-bugs-ness

Improve this answer
2

Defect density measures the number of defects found per unit of code measurement (never ever use lines of code for this), for a given period of time. The higher the number, the greater the density of bugs.

The density of bugs can be taken to be an inverse measure of the quality of the app. The lower the density (ie, the less defects reported), the higher the quality of the code.

But being a simple metric it has flaws:

  • If I simply do not test the code, my defect density is zero: perfect quality! Of course in reality, most untested code is far from good quality.
  • Conversely, for the same piece of code, I would get way more defects reported by super-zealous testers than I would for a lacklustre dev asked to do some testing. So I now have two densities (and thus two quality measures) for the one piece of code. And since I'm judging those testers on the number of defects raised, I'm incentivising them to score the quality as low as possible.
  • And of course, there's the issue of early testing. Pair a developer and tester during a sprint, and the output will likely be low in bugs. But it's extremely unlikely that the defects they found during the sprint will be recorded, so I've no measure of quality unless I retest just to gain my metric.

Defect density is just a metric. To be able to read more into it (quality of code, effectiveness of testing, likelihood of the app containing significant bugs etc) requires a heavy dose of subjectivity. Unless you know how effective your testing is, defect density won't be a reliable quality measure for example. But if you can't use metrics to measure effectiveness of testing, how do you measure it? So a subjective measure is needed.

Improve this answer
7
  • Yes but I still do not think I measure effectiveness of testing (as I cannot know whether there were 0 issues and hence 0 found, or 100 issues and all discovered and fixed). I think it reflects the maturity of dev-QA. Commented Oct 8, 2018 at 13:37
  • 1
    If I simply do not test the code, my defect density is zero: perfect quality! Of course in reality, most untested code is far from good quality. - That is not correct. If you have field reported issues, then you have a defect density of more than 0. Also, not tracking the issues doesn't mean they don't exist. Commented Oct 8, 2018 at 14:39
  • Yes, lines is a very imperfect measure: While drawn-out code likely has fewer bugs per line (it does little per line, so has little per line to do wrong), it probably has more bugs per unit due to monotony. Commented Oct 8, 2018 at 15:43
  • David, if the teams use the same languages, isnt KLOC still more accurate than story points? If there are coding guidelines and conventions, the differente should not be that bad, in my opinion. Commented Oct 9, 2018 at 6:14
  • @JohnV. There are two problems with using KLOC. Firstly, the more lines I write, the lower the density for a given number of defects. So if you are using defect density to measure my app's quality, it makes sense for me to make my code as verbose as my coding standards allow to make my app appear to be of higher quality. Secondly, unless you have offensively restrictive coding standards that remove all self expression in code, different devs will use different solutions to the same problem. 2-3 times difference in the line count is not unlikely there. So you aren't comparing fairly with KLOC. Commented Oct 9, 2018 at 9:39
1

If you get defect density from errors reported in shipped products, then you measure defects multiplied by customer willingness to engage with you. So if you fix the problems, and inform the reporters then they see that problems are solved, engagement goes up, defect reports go up, and since you fixed actual noticeable bugs your devs will learn to improve their development processes. At the same time your QA people learn what kinds of bugs they missed and improve their QA process.

So it’s important to see reported bugs as a chance to improve the product - while some companies make it their goal to shut up bug reports which is a terrible waste.

Next you should measure the average time from starting to look at a bug report from QA until the bug is reproducible. That time can be long, and it is wasted. If the amount is too much, allocate more time to creating bug reports. It is very helpful if QA can demonstrate in person how to reproduce a bug. This avoids time wasted by misunderstanding a bug report, and by bugs that are not reproducible because QA used slightly different steps in testing.

Then look at the number of trivial bugs found whenever new code is handed to QA. Decide if it would be more effective to use some more time in dev to produce code with fewer bugs, saving time for QA / describing bugs / reproducing and fixing them, plus missing some bugs. I have seen code that wasn’t just untested, it wasnt even run once. Which is a waste of everyone’s time.

Improve this answer

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.