A typical real computer system has an extremely large input space and no testing method can cover more than an infinitesimal part of it. On the other hand, broad regions of the input space will trigger bugs. The problem is that we do not know the shapes or locations of the buggy parts of the space. Random testing is a shotgun approach to finding these regions; empirically, it works well.
A few more observations:
- If the probabilities are chosen poorly, the tester will be far less effective than it could have been, and may not find any bugs at all
- Probabilities alone are an insufficient mechanism for engineering good test cases, and need to be combined with other mechanisms such as heuristic limits on the size of various parts of a test case
- Equivalent to the previous bullet, probabilities sometimes want to be a function of the test case being constructed; for example, the probability of creating a new function may decrease as the number of generated functions increases
- The connection between probabilities and high-level properties of test cases may be very indirect
- Alternatives to playing probability games, such as uniform sampling of the input space or bounded exhaustive testing, are generally harder or less effective than just getting the probabilities right
In practice, probability engineering looks like this:
- Think hard about what kind of test cases will drive the system under test into parts of its state space where its logic is weak
- Tweak the probabilities to make the generated tests look more like they need to look
- Look at a lot of generated test cases to evaluate the success of the tweaking
- Go back to step 1
Random testing almost always has a big payoff, but only when combined with significant domain knowledge and a lot of hard work.