Selection bias in sampling

Selection bias in sampling

   When evaluating research, critical analysis skills are essential in determining whether the decision that has been drawn is valid for a given sample, and most particularly when deciding whether the finding from a given study can be generalised to the wider population. Its something that many people either don't consider, or choose to ignore. Particularly when the conclusion drawn suits your ideological world-view. Especially then.

   Take for example the classic scientific scenario of a controlled trial. One group is given the treatment (which may be a drug, it may be a psychological intervention, or a different environment and set of stimuli). One group does not, we refer to this as the control group. If we compare the performance of the two groups at some future point in time, then any difference that is observed can be inferred to be the direct cause of the treatment. If we randomly allocate particpants to either the treatment and control, and both groups are equivilent at the beginning, then the treatment must be the cause!

Simple enough so far? Now imagine that instead of a randomized control trial, you allow the participants to choose which group they'd like to be in. For example, a group of clinically depressed people are offered the choice of an proven an efective anti-depressant, or some psychological counselling. We find that the group receiving the drug shows greater improvement than the counselling group. Does that mean that antidepressants work better than counselling?
Well it kind of depends.

Even when the groups are equivilent in age, gender mix, sociodemographic factors, etc. the problem is that the two groups might differ on some unobserved characteristics. Do you think that people who have a preference for medication might have some preconceived notions about its effectivness? Conversely, those who opt for counselling might have greater faith in psychology than psychiatry. When the participants choose, you cannot assume that unobserved characteristics won't come into place.

Let's illustrate it with another example. Selection bias is the problem facing educators and cognitive researchers in evaluating the influence of single-sex schooling over traditional coeducational schooling. In a recent consensus statement published in Science titled "The pseudoscience of single-sex schooling", Halpern et al., (2011) argue that the gold standard for evaluating single-sex schooling is a randomized controlled trial. But as critics of the article argue, who could imagine forcing such a choice on parents and children? A toss of a coin - heads you are coeducational, tails you go to a single-sex school. All we can do is look at the abundance of evidence that suggests making gender salient (such as emphasing the difference between boys and girls, or placing children in a same-sex learning environment) emphasises sex-typing and has the opposite of the intended effect. It makes boys and girls more sex-typed, rather than less.


Halpern, D. F., Eliot, L., Bigler, R. S., Fabes, R. A., Hanish, L. D., Hyde, J. S., et al. (2011). The pseudoscience of single-sex schooling. Science, 333(6050), 1706-1707. doi: 10.1126/science.1205031