Hypothesis testing

Via Sandwalk, we learn that psychology professor Irene Pepperberg has somehow changed her mind about the "classic scientific method" of hypothesis testing. It's not clear whether she herself chose the title, "The Fallacy of Hypothesis Testing"; the article discusses at best the limitations of hypothesis testing, not its fallacy. It's also possible she's discussing the fallacy that hypothesis testing is the sine qua non of the scientific method. But Dr. Pepperberg is not at all clear what she's actually changed her mind about.

She cites three criticisms of hypothesis testing: The scientific method doesn't tell us how to generate hypotheses, interesting questions can't be reduced to a single hypothesis, and many scientists don't appear to actually test their hypotheses. The last criticism is transparently specious: it's not a criticism at all of a method itself to observe that some people don't employ that method. (And, without actual evidence, it's impossible to determine whether her observation is even accurate. Any scientific study which employs the ubiquitous t-Test is at least comparing its stated hypothesis against the null hypothesis.)

Her criticism also that interesting scientific questions don't reduce to a single hypothesis also seems specious, although less obviously so. Her example,

"Can a parrot label objects?" may be a testable hypothesis, but actually isn't very interesting…what is interesting, for example, is how that labeling compares to the behavior of a young child, exactly what type of training might enable such learning and what type of training is useless, how far can such labeling transfer across exemplars, and….Well, you get the picture…the exciting part is a series of interrelated questions that arise and expand almost indefinitely.

decomposes a her "uninteresting" question to a number of interrelated hypotheses, each of which can be tested. She justly criticizes an absurd degree of "eliminative reductionism", but misses the target of hypothesis testing in general. Scientific theories are not themselves hypotheses, they are composed of hypotheses.

If Dr. Pepperberg is just now coming to the realization that the scientific method does not specify how to generate hypotheses, she has missed an important theme in philosophy beginning (at the latest) in 1974 with Pirsig's Zen and the Art of Motorcycle Maintenance. But again her criticism misses the mark. It's important to note that, while the generation of hypotheses is an interesting philosophical question, it has nothing to do with the scientific method. You may generate hypotheses any way you please, by intuition, bibliomancy, dreams, picking words out of a hat; you may even "simply to sit and observe and learn about one's subject before even attempting to devise a testable hypothesis." The scientific method just specifies that you have to express what you're testing in a particular form, a testable hypothesis, and then you have to test it. One can form a lot of potentially useful ideas from unbiased (or at least open-minded) observation, but to make it science — to turn these ideas into knowledge — you need to express your ideas as hypotheses, and test them.

It's not even possible (or particularly useful) to "observe [a] system without any preconceived notions," and it is not possible by such observation to actually gain knowledge. The first thing a good scientist should do after this sort of observation would be to figure out how the preconceived notions she wasn't previously aware of might have biased the observations. She has to assume it's possible that everything she thinks she then "knows" might be incorrect. She must then formulate testable hypotheses and test them.

To her credit, Dr. Pepperberg doesn't abandon the notion of hypothesis testing itself. She clearly describes reducing a scientific question to a collection of testable hypotheses, and clearly does not advocate simply stopping at unbiased observation as good enough in itself to generate real knowledge. At best, she has changed her mind only about an overly restrictive formulation of the scientific method employed far more often by philosophers seeking a straw man to demolish than actual working scientists.