Next / Previous / Contents / Shipman's homepage

11. Testing


Program testing can be used to show the presence of bugs, but never to show their absence!

 -- Edsger J. Dijkstra, Comm. ACM 15(10), Oct. 1972: pp. 859–866.

If you aren't convinced the code is correct before you start testing, what could possibly convince you?

 --Donald E. Grimes (pers. comm.)

Once the entire design has been peer-reviewed (or self-reviewed with trace tables if peers are unavailable), testing proceeds by these rules.

An example of such a list is the one for the author's nomcompile project.

We can never know how many defects remain in a system. However, there is one empirical test that can increase (or decrease) our confidence. If the frequency of defect discovery decreases sharply after the initial testing phase, it suggests that the defect discovery rate is asymptotic to the x-axis. Conversely, if the rate of discovery of defects decreases in a more linear fashion, it suggests that there are many more defects to be discovered before the defect count approaches zero.

11.1. Treasure your mistakes, don't bury them!


Good judgment comes with experience, but experience comes from bad judgment.

 --Attributed to Mulla Nasreddin

When you find a defect, there are two responses. The first is to fix the defect.

If you aspire to mastery, your second response is to look at why the defect occurred. Many arise from sheer carelessness. However, it is worthwhile to give some thought to how you might modify your approach to prevent this sort of defect from happening again.

In particular, if you use a particular trick or pattern and find a defect in it, make sure you look at all the other places you have used that same trick or pattern. You may find that they are broken as well.