sacrifice and then inverts the order of his follow-up moves or misses the really clinching point of his combination.” [Reinfeld, 1959]
When the exorcist arrived at the house, he almost immediately started upstairs to begin the exorcism. “Wait,” interrupted the attending priest, “Don’t you want to learn the personalities of the demons?” “There is only one,” replied the exorcist. [Blatty, 1972]
Many of the potential pitfalls to optimum experimental design are obvious from earlier parts of this chapter, particularly the section, ‘Tips on Experimental Design and Execution’. Most of these pitfalls, however, are manifestations of the same demon: a rogue, or uncontrolled, variable.
Control of Variables
Rogue variables are a frequent scientific problem. Suspect such a problem when troubleshooting equipment or an experimental setup, if none of the initial troubleshooting techniques helps. Also suspect such a problem whenever an experiment gives surprising, unexpected results. Such problems are always a nuisance, but sometimes their solution can foster scientific insight.
“The notion of a finite number of variables is an idealization” [Wilson, 1952] that is essential to practical science. Most ‘relevant’ variables have only a trivial influence on the phenomenon of interest. Often, they have no direct causal relationship to this phenomenon or variable, but they do have some effect on one of the primary causal variables. Such variables are second or third-order problems that are ordinarily ignored. Usually the scientific focus is on identifying and characterizing the primary causal variables -- those that have the greatest influence on the phenomenon of interest.
In the so-called ideal experiment, the investigator holds all relevant variables constant except for a single variable. This independent variable is deliberately varied while measuring the resulting changes in a dependent variable. Simplicity gives power to such experiments, but they are based on the often dubious assumption that one knows all relevant variables. Usually, we hold as many relevant variables constant as possible and cope with the non-constant variables through randomization. Unfortunately, the variables that we can control are not necessarily the ones that are most important to control.
In Chapters 2 and 3, we considered statistical techniques for quantitatively estimating the influence of variables. Here the focus is on several methods for determining whether or not a variable is crucial. Selection of the most appropriate procedure depends on feasibility and on time and effort needed to remove or measure a variable.
Common techniques for dealing with a problem variable are:
- stabilization: Keeping a variable constant prevents it from influencing other variables. This approach is best for variables that are a disruptive influence (e.g., voltage or temperature variations), rather than scientifically interesting. Rarely, it is feasible to monitor the problem variable, then make measurements only when it has a certain value. The technique does not work for intermittent problems.
