Selection bias in sampling

Selection bias in sampling

   When evaluating research, critical analysis skills are essential in determining whether the decision that has been drawn is valid for a given sample, and most particularly when deciding whether the finding from a given study can be generalised to the wider population. Its something that many people either don't consider, or choose to ignore. Particularly when the conclusion drawn suits your ideological world-view. Especially then.

   Take for example the classic scientific scenario of a controlled trial. One group is given the treatment (which may be a drug, it may be a psychological intervention, or a different environment and set of stimuli). One group does not, we refer to this as the control group. If we compare the performance of the two groups at some future point in time, then any difference that is observed can be inferred to be the direct cause of the treatment. If we randomly allocate particpants to either the treatment and control, and both groups are equivilent at the beginning, then the treatment must be the cause!

Simple enough so far? Now imagine that instead of a randomized control trial, you allow the participants to choose which group they'd like to be in. For example, a group of clinically depressed people are offered the choice of an proven an efective anti-depressant, or some psychological counselling. We find that the group receiving the drug shows greater improvement than the counselling group. Does that mean that antidepressants work better than counselling?
Well it kind of depends.

Even when the groups are equivilent in age, gender mix, sociodemographic factors, etc. the problem is that the two groups might differ on some unobserved characteristics. Do you think that people who have a preference for medication might have some preconceived notions about its effectivness? Conversely, those who opt for counselling might have greater faith in psychology than psychiatry. When the participants choose, you cannot assume that unobserved characteristics won't come into place.

Let's illustrate it with another example. Selection bias is the problem facing educators and cognitive researchers in evaluating the influence of single-sex schooling over traditional coeducational schooling. In a recent consensus statement published in Science titled "The pseudoscience of single-sex schooling", Halpern et al., (2011) argue that the gold standard for evaluating single-sex schooling is a randomized controlled trial. But as critics of the article argue, who could imagine forcing such a choice on parents and children? A toss of a coin - heads you are coeducational, tails you go to a single-sex school. All we can do is look at the abundance of evidence that suggests making gender salient (such as emphasing the difference between boys and girls, or placing children in a same-sex learning environment) emphasises sex-typing and has the opposite of the intended effect. It makes boys and girls more sex-typed, rather than less.


Halpern, D. F., Eliot, L., Bigler, R. S., Fabes, R. A., Hanish, L. D., Hyde, J. S., et al. (2011). The pseudoscience of single-sex schooling. Science, 333(6050), 1706-1707. doi: 10.1126/science.1205031

     While the intention of this blog will be to provide useful and insightful commentary about the field of applied psychology (and from time to time, science in general), its main goal is to help me organise my thoughts and notes about the field as I continue my PhD.  PhD's are tricky beasts, and can easily morph off-topic (in computer science terms, feature creep). While I'm studying gender psychology, its very much a cross-disciplinary field with influences from education, cognitive psychology, psychometrics, and cross-cultural studies.

     So sometimes I'll use this blog to write a short piece about these fields, either for my own personal interest or perhaps with vague notions of turning it into a peer-reviewed publication output at some distant point in the future. Us academic types love publication outputs, even if they're in the most obscure and least read journal on the planet. Especially if they're in an obscure and less read journal. Why? because they're usually subject to less scrutiny, making it easier to get some outrageous claim or theory into the body of literature. There's a bias in science literature to accepting anything - even if it is far-fetched, untested, and on occasion indefensible - if its published in a peer-reviewed journal. Because peer-review weeds out anything dodgy, and can then be cited ad infinitum by others. And often by yourself in future publications. Caveat emptor, and all that.