Programmer Musings

Accuracy and Precision

When I was getting my EE degree many years ago, one of my professors (Dr. Dave) had an interesting lecture on the difference between accuracy and precision. Even though many people use these terms interchangeably, they are separate concepts. Possibly the most important point of separating these two concepts is remembering that they may be independent. The following diagram shows the difference between accuracy and precision (and is taken from my memory of the lecture).

On this chart, accuracy increases as you move upward and precision increases as you move to the right. The upper right corner represents what most people think of when they are talking about either high accuracy or precision. The lower left corner represents what people think of as low accuracy or precision. Interestingly, most people don't think about the other two quadrants.

As programmers, we deal in words and concepts all of the time. There is almost nothing physical about most of what we do. This means that concepts are extremely important to our work. Subtle distinctions in concepts can mean the difference between a project that works and one that fails. In this case, differentiating between precision and accuracy can be quite useful. For example, if you are not accurate, it doesn't matter how precise you are (lower right corner). This is obvious when looking at the bullseye in the picture, but most of us have seen projects where someone tries to fix an accuracy problem by changing from float to double to get more decimal places. Changing data types may change the outcome due to a change in dynamic range, but it usually is not enough to just change the number of significant digits in the output.

Likewise, if an experiment is not giving the right answer, increasing the number of measurements may not change the outcome. More data to average increases the precision, but does not change the accuracy of the answer. Understanding these differences can mean the difference between wasting a lot of time generating a very precise, wrong answer and rethinking an approach to increase the accuracy of the answer.

Amusingly enough, we have all usually run into cases like the upper left corner. We know we have data in the right general area (accurate), but any individual point may vary due to some kind of noise. By averaging a large number of points we increase the precision without affecting the accuracy. Although most of us have a good feel for the mechanics of averaging, making the distinction between precision and accuracy simplifies the discussion of what we are doing. Which leads us to an important observation, simple post-processing of data can increase the precision of an answer, but it rarely changes the accuracy.

Throughout our field, concepts like these mean the difference between making progress and wasting time. Sometimes the differences between concepts are subtle, but that does not make them less important.

Review of Waltzing with Bears

Waltzing with Bears
Tom DeMarco & Timothy Lister
Dorset House Publishing, 2003

The authors of Peopleware are back to tackle the topic of risk management. Given their earlier works, you would expect DeMarco and Lister to provide good insights into the topic and clear explanations. In that, you will not be disappointed with this book.

The book explains why you should do risk management, how to do risk management, and how to test the management you are doing. Surprisingly, the book also tells you when you might not want to do risk management and some of the politics that can surround this topic. The authors describe many facets of some corporate cultures that work to defeat the management of risk. They also explain why taking risks is critical to success and how to judge acceptable risks from foolhardy ones. As you would expect, the authors are clear in their explanations and instructions.

The book references an accompanying website that lists various risk management tools including an risk simulation tool supplied by the authors. This tool may be used to map the uncertainty in your project's estimates. Understanding this uncertainty is the heart of risk management.

The only complaint I have with the book is in the area where I feel I am the weakest. The authors acknowledge that many of the problems in risk management are problems with corporate culture or attitudes of the various stakeholders in the project. Unfortunately, their advice on dealing with these issues is extremely vague compared to the more technical aspects of risk management. I understand that this is the part that is the hardest to quantify, but it may be where we need the most advice. I suppose that actual project managers may have a better idea of how to deal with these issues and do not need more instruction. But, for me, this seemed to be an area where the book was lacking.

In general, I feel this is a great book for anyone who must manage a project or who needs to track the risks on a project. If your focus in most projects is purely in the area of code, you may not get as much out of the book. However, if you need to lead a team or manage a project, this book is a great way to become familiar enough with risk management to possibly succeed. I would definitely recommend the book to senior developers and junior to intermediate project managers. More senior project people may already be aware of the issues covered in the book, but may benefit from the fresh treatment.

Diff Debugging

Every now and then, I manage to pull myself away from reading and reviewing computer books (my hobby for the last year) or programming (my hobby for ... never mind), and spend a little time on various weblogs. It's important to see what the names in the field are thinking and talking about.

During one of these late night blog-reading sessions, I ran across MF Bliki: DiffDebugging. This turned out to be a wonderful find, not because the technique was new to me (I've been using it almost as long as I've used version control), but because he gave the technique a useful name.

Fowler explains it much better than I do, of course, but the technique is great for finding regressions in a code base. If you detect a bug in some code you are working on that you know worked before, use your version control software (VCS) to find a version that did not manifest the bug. Then, try to find the last version that did not have the bug and the first version that did. (These should be adjacent versions.) By only looking at what changed between these two versions, it may be possible to find the cause of the bug much more quickly.

I have often used the technique with dates instead of individual commits. This is the easier approach if your VCS does not support atomic commits (CVS, RCS, etc.). If your VCS does support atomic commits (Subversion, etc.), you can test individual commits, which may simplify focusing in on the change.

If you have good unit tests for your software, diff debugging is even faster. You can just run the unit test which shows the bug for each version that you check out. Success and failure are much easier to detect. However, formal unit tests are not required to use diff debugging, since I was using the technique long before I became test-infected. Automated tests do make the technique much easier, though.

One important point that Fowler does not make is that a binary search technique can be pretty useful if the range of versions is large. If you know the code worked a month ago and it doesn't work now, a good approach would be:

1. Check out the code from a month ago and verify that it actually worked.
2. Check out the code from two weeks ago and see if it worked.
   • If it works, check out code from a week later to see if it works.
   • If it doesn't, check out code from a week earlier to see if it works.

Repeat this technique with shorter time scales until you find the offending version. Of course, when you are reduced to a handful of versions, it is easier to just step through them (as in Fowler's example). I have often needed to the binary search technique in cases where a standard build and smoke-test procedure was not in place.

The most interesting part of Fowler's blog entry was the fact that he created a good name for this technique. If you don't have a name for something, it is really hard to talk about. It is also not as obvious a tool in your toolkit. By giving the technique a name, Fowler has improved my programming skills by turning an ad hoc technique I have applied in certain situations into a known tool in my programming toolkit. The technique has not changed, but the name makes it a more reusable tool.

This is an interesting fact about our field: concepts are our best tools. Naming a concept gives you the power to use it. Many of the most important breakthroughs in the past few decades have not been new techniques or algorithms, but the naming of techniques so that we can discuss them and reuse them.