effects are being detected (e.g., card guessing tests for extrasensory perception, or ESP), much research has investigated the phenomenon of ‘motivated scoring errors.’ Scoring hits or misses on such tests appears to be objective: either the guess matched the card or it did not. Mistakes, however, are more subjective and biased: believers in ESP tend to record a miss as a hit, and nonbelievers tend to score hits as misses.
Parapsychology experimenters long ago adapted experimental design by creating blinds to prevent motivated scoring errors, but researchers in most other fields are unaware of or unworried by the problem. Motivated scoring errors are subconscious, not deliberate. Most scientists would be offended by the suggestion that they were vulnerable to such mistakes, but you and I have made and will make the following subconsciously biasing mistakes:
- errors in matching empirical results to predictions,
- errors in listing and copying results,
- accidental omissions of data, and
- mistakes in calculations.
• missing the unexpected: Even ‘obvious’ features can be missed if they are unexpected. The flash-card experiment, discussed earlier in this chapter, was a memorable example of this pitfall. Unexpected results can be superior to expected ones: they can lead to insight and discovery of major new phenomena (Chapter 8). Some common oversights are: (1) failing to notice disparate results among a mass of familiar results; (2) seeing but rationalizing unexpected results; and (3) recording but failing to follow-up or publish unexpected results.
• biased checking of results: To avoid mistakes, we normally check some calculations and experimental results. To the extent that it is feasible, we try to check all calculations and tabulations, but in practice we cannot repeat every step. Many researchers selectively check only those results that are anomalous in some way; such data presumably are more likely to contain an error than are results that look OK. The reasoning is valid, but we must recognize that this biased checking imparts a tendency to obtain results that fulfill expectations. If we perform many experiments and seldom make mistakes, the bias is minor. For a complex set of calculations that could be affected substantially by a single mistake, however, we must beware the tendency to let the final answer influence the decision whether or not to check the calculations. Biased checking of results is closely related to the two preceding pitfalls of making motivated mistakes and missing the unexpected.
• missing important ‘background’ characteristics: Experiments can be affected by a bias of human senses, which are more sensitive to detecting change than to noticing constant detail [Beveridge, 1955]. In the midst of adjusting an independent variable and recording responses of a dependent variable, it is easy to miss subtle changes in yet another variable or to miss a constant source of bias. Exploitation of this pitfall is the key to many magicians’ tricks; they call it misdirection. Our concern is not misdirection but perception bias. Einstein [1879-1955] said, “Raffiniert is der Herrgott, aber boshaft ist er nicht” (“God is subtle, but he is not malicious”), meaning that nature’s secrets are concealed through subtlety rather than trickery.
• placebo effect: When human subjects are involved (e.g., psychology, sociology, and some biology experiments), their responses can reflect their expectations. For example, if given a placebo (a pill containing no medicine), some subjects in medical experiments show a real, measurable improvement in their medical problems due to their expectation that this ‘medicine’ is beneficial.
