An undergraduate student of mine recently showed me a scientific journal article he had been reading that confused him. Specifically, one result of the paper was that people who ate a vegetarian diet had higher cancer rates than those who ate meat [1]. He was confused by this because he said he had read other papers that suggested the exact opposite [for example 2, 3]. The point of this post is not to get into any discussions about the health effects of meat, but rather to briefly discuss the implications of contradictory results from the scientific literature. There are three issues that are immediately relevant and fairly straightforward for a growing scientist or a non-scientist: variation in how studies are carried out, the ‘random sample’-based idea of science, and reproduction of research.

Perhaps the simplest of these is the idea that if you ask the same question in different ways, you might find different results. I would personally explain this by pointing out that how you ask the question actually changes the question itself, just a little bit. To illustrate this, start with the question, “Does eating meat increase your risk of cancer?” To test this, one might start with a group of people who eat the average omnivorous American diet, remove all meat from their diets, and compare their cancer rates to the control (meat-eating) group five years later. This study has in fact modified the question slightly and asked, “How does removing meat from an omnivorous diet affect cancer risk?” Alternatively, if you compare cancer rates in the general population between vegetarians and omnivores, you are asking, “How do cancer rates compare among people who eat vegetarian diets compared to those who eat omnivorous?” At first glance, this may seem a pedantic distinction, but in practice this difference can be major. Can you think of any reasons people might not eat meat that could affect their health in other ways? What about people who have a major family history of cancer? They might be very careful about what they eat, but might also have an elevated risk of cancer even if they take special care.

The second issue here is the ‘random sample’-based idea of science. In essence, scientists are trying to make general conclusions that are generalizable to all people, not just those in their current study. In order to do that, they often try to collect a sample of people that is representative of the entire population. If they fail to do that, it’s not strictly a flaw in study, but it becomes important to be clear about which populations the results can be generalized to. For example, imagine the only vegetarians I can find for a research study are part of specific religions such as Buddhists or Seventh-day Adventists, along with a smattering of upper-middle class white Lutherans, pretty common in my part of the country. Well, in the end, that’s a pretty limited sample of people. What else might those populations do differently than the general population?

Finally, science is based upon the idea of repeating the same study many times. Both due to the random sample idea, but also the way we do statistics to come to conclusions, there is a certain possibility of error to every study. A well-designed and well-conducted study with a fairly large sample size, carried out by objective researchers who have no financial stake in the outcome, should have a low possibility of simply being ‘wrong.’ Unfortunately, every individual study does have this possibility. Studies in which the researchers have a heavily vested interest, or that are carried out by people who are not careful and objective, may have a higher likelihood of making errors. Now, this may seem depressing: “Why should we even believe science?” Nevertheless, the true power of science arises in reproducibility. If ten different researchers have conducted the exact same research, and nine reach the same conclusions, you can be pretty confident in the results of the nine. To, be clear, the contradictory study may be of very high quality as well, and may simply have found a different result due to random chance.

So, when you read a study, and you find the results confusing, or contradictory to your prior beliefs, here is my suggestion. First, keep an open mind. Maybe your prior beliefs were wrong! Approach every new idea considering the possibility that it might be right. Second, think critically about the study. The authors have probably stated a research question or hypothesis. But ask yourself, is their stated question really the question they asked with their research? What population do they say they studied? Did they really? Finally, look for more. Do they cite other related research? What did that prior research conclude? Has any new research cited the paper in question? Try using Google Scholar, or any number of other search tools to find other research on the same topic. Look for a review! Some articles are actually just summaries of many other research articles, and can be enormously helpful in this situation. Maybe yours is the only relevant research, but maybe there is a large body of science either supporting it, or failing to. Happy sciencing!

References

1. Burkert, N.T., et al., Nutrition and Health – The Association between Eating Behavior and Various Health Parameters: A Matched Sample Study. PLoS ONE, 2014. 9(2): p. e88278.
2. Tantamango-Bartley, Y., et al., Vegetarian Diets and the Incidence of Cancer in a Low-risk Population. Cancer Epidemiology Biomarkers & Prevention, 2013. 22(2): p. 286-294.
3. Key, T.J., et al., Cancer incidence in vegetarians: results from the European Prospective Investigation into Cancer and Nutrition (EPIC-Oxford). The American Journal of Clinical Nutrition, 2009. 89(5): p. 1620S-1626S.