Program testing can be used to show the presence of bugs, but never to show their absence!
|-- Edsger J. Dijkstra, Comm. ACM 15(10), Oct. 1972: pp. 859–866.|
If you aren't convinced the code is correct before you start testing, what could possibly convince you?
|--Donald E. Grimes (pers. comm.)|
Once the entire design has been peer-reviewed (or self-reviewed with trace tables if peers are unavailable), testing proceeds by these rules.
No compilation or testing of any kind during design. No unit testing.
Build a suite of regression tests that test as many overall features of the system as you can think of. The idea is to use your regression suite for initial testing; later, as the design is modified for additional features, you can use test to make sure that each new change did not break any existing features.
A good regression suite will print, at the very end, some kind of summary such as “Number of tests failing: 2”. Thus, even if the suite produces a lot of output, you can look at the end of the report to see if any defects were detected.
Build a list of all defects discovered since the first compilation. Break this list up into two or three categories:
Pure logic defects: you did the wrong thing. Example: a
“<=” where there should have been a
Syntax defects: anything flagged by the language processor.
For languages such as Python that do their type-checking at execution time, there is a third category: defects that would have been detected in a more strongly typed language.
An example of such a list is the one for the author's nomcompile project.
We can never know how many defects remain in a system. However, there is one empirical test that can increase (or decrease) our confidence. If the frequency of defect discovery decreases sharply after the initial testing phase, it suggests that the defect discovery rate is asymptotic to the x-axis. Conversely, if the rate of discovery of defects decreases in a more linear fashion, it suggests that there are many more defects to be discovered before the defect count approaches zero.
Good judgment comes with experience, but experience comes from bad judgment.
|--Attributed to Mulla Nasreddin|
When you find a defect, there are two responses. The first is to fix the defect.
If you aspire to mastery, your second response is to look at why the defect occurred. Many arise from sheer carelessness. However, it is worthwhile to give some thought to how you might modify your approach to prevent this sort of defect from happening again.
In particular, if you use a particular trick or pattern and find a defect in it, make sure you look at all the other places you have used that same trick or pattern. You may find that they are broken as well.