Pitfalls in Observational Research

So far observational research sounds good. It is relatively easy (all you do is collect data) and if one is fortunate enough to find strong correlations in the data, one can make powerful predictions. Now we return to the theme of critical thinking. It is time to discuss some of the pitfalls or things that can go wrong when interpreting observational research.

Population validity problems

What does it mean if a finding has "low population validity"?

One big problem is population validity. Does data gathered from one popuation apply accurately to other populations? This cannot be assumed. We encountered a population validity problem when discussing the relationship between level of alcohol consumption and low grades. A strong correlation existed at the University of Illinois but a rather weak one at Virginia Tech. There might not be any correlation at all in France. A finding that is not accurate when applied to a new population is said to have low population validity.

What does it mean to "generalize"?

The issue of population validity is potentially relevant whenever one generalizes between two groups. To "generalize" means to treat two different things the same way. Generalizing between groups means assuming results obtained with one group will be accurate when applied to another group. If the groups are different in some important way, this assumption might be false. In that case, predictions based on data from one group might not be accurate when applied to the other group.

What is a problem with tests of cancer-causing substances?

A potential population validity problem occurs when researchers generalize from animals to humans. For example, some species of rats are 10,000 times more likely to grow cancers than humans. Therefore tests of cancer-causing substances using rats as subjects may have low population validity for humans.

How did research on an alcohol-nullifying drug show the importance of population validity?

In the mid-1980s a new drug was reported to make drunken animals behave as if sober by blocking the behavioral effects of alcohol intoxication. Pictures of rats showed them drunk before taking the drug, sober afterward. There were no bad side effects. Researchers were excited by this, because such a drug could be extremely useful. For example, Emergency Room physicians could use it to revive humans suffering from acute alcohol intoxication. Unfortunately, further research showed the drug caused tremors and convulsions in monkeys, indicating that it probably was not safe for humans, whose nervous systems resemble that of a monkey more than that of a rat. Miczek and Weerts (1987) pointed out that this reaffirmed the importance of testing new drugs on monkeys and apes, not just on rodents, before trying them on humans.

What sorts of population differences can occur with different groups of humans? How can the problem be dealt with?

Population validity problems can also occur when generalizing between any two groups of humans. Data collected in a door-to-door survey may not hold true for the population as a whole. Results collected in Germany may not apply to people in France. Data collected using a subject pool that draws students from introductory psychology classes might not be comparable to data that draws students from business classes

For this reason, one must be clear in a research report about where and how data was collected. If the target population is well defined (for example, one is interested only in business majors at a particular school) then one must try to obtain a random sample of that particular population. If the target population is very general, for example, one is trying to prove something about all humans, then conclusions based on a one group of subjects must be regarded as tentative until the research can be replicated with different groups, to see if they might react differently.

How can differences between groups be informative to researchers?

Sometimes differences between groups of people are interesting in themselves. Researchers who study abnormal psychology find it significant when a mental disorder occurs in only one culture, not others. This suggests that the disorder may involve a belief system or genetic or environmental condition peculiar to that culture. On the other hand, if a mental disorder (like schizophrenia or depression) occurs in every culture, that suggests it is due to genes or conditions common to all humans.


Write to Dr. Dewey at psywww@gmail.com.

Don't see what you need? Psych Web has over 1,000 pages, so it may be elsewhere on the site. Do a site-specific Google search using the box below.

Custom Search

Copyright © 2007 Russ Dewey