Hypothesis testing does not always require quantification and the other trappings of modern science. ... It should be possible to use [descriptive data about religion] to evaluate the major evolutionary hypotheses. ...Wilson does not correctly use statistical methodology; he presents statistical concepts in a way that harms rather than helps the reader's statistical intuition.
Of course, it is necessary to gather the information systematically rather than picking and choosing examples that fit one’s pet theory. In Darwin’s Cathedral, I initiated a survey of religions drawn at random from the 16-volume Encyclopedia of World Religions, edited by the great religious scholar Mircia Eliade. The results are described in an article titled “Testing Major Evolutionary Hypotheses about Religion with a Random Sample,” which was published in the journal Human Nature and is available on my website. The beauty of random sampling is that, barring a freak sampling accident, valid conclusions for the sample apply to all of the religions in the encyclopedia from which the sample was taken.
First, random sampling is about statistics, statistics are mathematical, and mathematics is about quantities, not qualities. You cannot use statistical methods on qualitative data. Period. Wilson proceeds to draw a quantitative conclusion from his method:
By my assessment, the majority of religions in the sample are centered on practical concerns, especially the definition of social groups and the regulation of social interactions within and between groups. [emphasis added]"Majority" is a quantitative term: It means at least 51% of the sample has some characteristic. One must count something to get a percentage. But if you're going to quantify something, it's not sufficient to quantify it by pure subjective assessment, and it's doubly worse to draw subjective conclusions about the sample as a whole and call them statistically quantified. You need to have an objective procedure to precisely specify what you're counting and how you're counting it.
Second, you cannot use random sampling or statistics to draw conclusions about the population from the sample; you can use random sampling only to estimate the statistics of the population from the statistics of the sample. The difference is subtle but important. I cannot simply pick out 100 people and then say that 1% of all people are exactly like each person in the sample. I don't think Wilson is actually doing something this obviously stupid, but he doesn't even mention what he's quantifying and how he's drawing his statistical conclusions.
Even if you quantify something, you have to ensure that your sample is large enough to draw conclusions about the population statistics. For instance, a random sample of 10 people from the New York City phone book would not be large enough to draw conclusions about the prevalence of given names. One can actually quantify the power of a sample with the aptly named technique of Power Analysis.
One might hope that the presentation in the eSkeptic article was merely an unfortunate oversimplification for a lay audience. However, an examination of the published scientific study, Testing Major Evolutionary Hypotheses about Religion with a Random Sample, reveals the same conceptual errors as in the article: A detailed analysis is given for the selection criteria, but there is absolutely no description whatsoever of the methodology used to actually analyze each religion; the analysis is purely subjective.
The paper itself is not all that bad, actually. There's nothing wrong with subjective analysis per se, and a random survey has real value in a new scientific field. But the use of statistical language is misleading, and adds a purely pseudo-scientific veneer to an otherwise interesting paper. Given the subjective nature of the analysis, it would have been sufficient to just say, "I picked 35 religions more-or-less at random and evaluated them subjectively, here's what I saw."
Far better, from a scientific perspective, would be to formulate some hypothesis precisely and state that, "If my hypothesis were false, we would see thus-and-such; we picked 35 religions more or less at random and saw no evidence of thus-and-such."