• weighting one’s own data preferentially: This problem is universal. We know the strengths and weaknesses of personally-obtained data much better than those of other published results. Or so we rationalize our preference for our own data. Yet both ego and self-esteem play a role in the frequent subjective decision to state in print that one’s own evidence supersedes conflicting evidence of others.
• failure to publish negative results: Many experimental results are never published. Perhaps the results are humdrum or the experimental design is flawed, but often we fail to publish simply because the results are negative: we do not understand them, they fail to produce a predicted pattern, or they are otherwise inconsistent with expectations. If we submit negative results for publication, the manuscript is likely to be rejected because of unfavorable reviews (‘not significant’). I have even heard of a journal deliberately introducing this bias by announcing that they will not accept negative results for publication. Yet a diagnostic demonstration of negative results can be extremely useful -- it can force us to change our theories.
• concealing the pitfalls above: The myth of objectivity usually compels us to conceal evidence that our experiment is subject to any of the pitfalls above. Perhaps we make a conscious decision not to bog down the publication with subjective ambiguities. More likely, we are unaware or only peripherally cognizant of the pitfalls. Social scientists recognize the difficulty of completely avoiding influence of the researcher’s values on a result. Therefore they often use a twofold approach: try to minimize bias, and also specifically spell out one’s values in the publication, so that the reader can judge success.
“The great investigator is primarily and preeminently the man who is rich in hypotheses. In the plenitude of his wealth he can spare the weaklings without regret; and having many from which to select, his mind maintains a judicial attitude. The man who can produce but one, cherishes and champions that one as his own, and is blind to its faults. With such men, the testing of alternative hypotheses is accomplished only through controversy. Crucial observations are warped by prejudice, and the triumph of the truth is delayed.” [Gilbert, 1886]
Pitfall Examples
Penzias and Wilson [1965] discovered the background radiation of the universe by accident. When their horn antenna detected this signal that was inconsistent with prevailing theories, their first reaction was that their instrument somehow was generating noise. They cleaned it, dismantled it, changed out parts, but they were still unable to prevent their instrument from detecting this apparent background radiation. Finally they were forced to conclude that they had discovered a real effect.
Pitfalls: biased checking of results; biased rejection of measurements;
Throughout the 20th century, scientists from many countries have sought techniques for successfully predicting earthquakes. In 1900 Imamura predicted that a major earthquake would hit Tokyo, and for two decades he campaigned unsuccessfully to persuade people to prepare. In 1923, 160,000 people died in the Tokyo
