BETA
This is a BETA experience. You may opt-out by clicking here

More From Forbes

Edit Story

How To Avoid Being Fooled By Health Claims: A Few Simple Rules

This article is more than 8 years old.

When it comes to health, every day can be April Fool’s Day. First we’re told to avoid meat and take vitamins. Now scientists are saying meat is okay and vitamin pills are useless and potentially bad for you. There are claims that you should eat dinner early to stay thin, others that you should eat many small meals. Scientists claim their studies show you should avoid fats, or eat more fats; that we should eat more protein, or cut back on protein.

When I’m investigating these kinds of claims, my scientific sources sometimes refer to something called “Hill’s Criteria of Causation” – a 9-step guide anyone can use to help evaluate a given claim, whether it’s about food, homeopathy, acupuncture, antidepressants or cold remedies. The criteria date back to 1965, when physician Austin Bradford Hill set them out in a paper. He pointed out that many studies may connect people who eat food X with people who enjoy health benefit Y, but it doesn’t necessarily follow that X causes Y.

The criteria can help non-scientists understand why the vast majority of scientists believe that smoking causes cancer and HIV causes AIDS, but don’t believe vaccines cause autism or that cell phones cause cancer. Why not just go with the consensus? That’s not a bad rule of thumb, but in some cases, scientists haven’t formed a consensus yet. Do diet sodas make you fat? That depends on which scientist you talk to. Should you eat a low carbohydrate diet to lose weight? Cut out salt? Experts remain divided on those questions too, so it helps to have the tools to evaluate claims yourself.

The best summary I could find on Hill's Criteria comes from Mark Crislip writing in the blog Science Based Medicine. Dr. Crislip’s emphasis is on investigating claims from alternative medicine, so his examples focus on things such as acupuncture and homeopathy, which tend to fail Hill’s tests. But more mainstream medical ideas can fail as well, especially in nutrition, where mainstream doctors can reach conflicting conclusions. Below are five of the nine rules of Hills I found most applicable to the kinds of studies promoted in the news.

Strength: When scientists claim they see an effect in a study, is the connection strong or weak? As an example of a strong effect, Hill uses studies that claimed an association between cancer of the scrotum and working as a chimney sweep. In that case, the cancer is 400 times more prevalent in chimney sweeps than in the general population. In contrast, Dr. Crislip says alleged evidence for alternative therapies tends to be weak:

The strength of ‘alternative’ therapies usually hovers around background noise, usually at the level of personal experience. If acupuncture or homeopathy were 400 times superior to placebo, there would no discussion of its validity.

As I found reporting this post for Forbes, the studies suggesting a connection between cell phone use and brain cancer also tend to be weak. And as I learned from psychologist Irving Kirsch, the effect of many antidepressants is also hovering in the noise.

Last month scientists announced a study that suggested diet sodas cause the accumulation of belly fat. As I discovered reporting on the study, it passed a common test known as statistical significance but it’s still a small, weak effect. Over a period of 10 years diet soda drinkers gained just a fraction of an inch more in waist circumference than did non-drinkers. Statistical significance is a measure of the likelihood of a result occurring by chance, but it says nothing about the actual significance of the association.

Consistency: Hill uses the association between cigarettes and cancer as a connection that shows up in study after study, and that gets stronger as more studies are done. Dr. Crislip points out that the opposite happens in tests of homeopathy and other popular alternative remedies, where the effect gets closer to zero as higher quality studies are done. The same has happened as scientists have looked into alleged connections between cell phone use and cancer.

In the mainstream arena, Dr. Kirsch found that the more studies that were done on antidepressants, the more they began to resemble placebos. Kirsch said that drug companies brush the lack of consistency under the rug by only publicizing the fraction of studies that show an effect, while hiding a bulk of studies in which the drugs failed to beat placebos.

That’s why the diet soda/belly fat study should be accepted with caution until more studies are done. Single studies can be misleading, especially when the effect is weak.

Mainstream dietary advice is also notorious for lacking consistency. Consistency is put to the test when scientists pull together multiple previous studies to produce a so-called meta-analyses. Then suddenly they announce that the vitamin pills our doctors told us to take do us no good and the fats they told us to avoid may do us no harm. There’s also no consistent evidence to back the assumed health benefits of being thin as opposed to carrying what the standard BMI charts would deem a few “extra” pounds. Some studies even suggest the mildly overweight are healthier than the skinny people.

Biological Gradient: Usually, when there’s something going on beyond a placebo effect, a bigger dose of something will result in a bigger effect. The more cigarettes people smoke, the more likely they are to get cancer, for example. Dr. Kirsch has pointed out that antidepressants don’t show a biological gradient, which is consistent with their working primarily through the placebo effect. The claims that cell phones cause cancer tend to lack any connection between the dose and the response.

The diet soda study, on the other hand, did show a larger belly fat gain in heavy diet soda drinkers than in those who indulged only occasionally. So it did pass one of Hill’s criteria.

Plausibility: In science, things work through ream mechanisms rather than by magic. Scientists may not understand how things work right away, but they tend to sort it out eventually if they can at least provide a plausible mechanism at the start.

Early on in the AIDS epidemic, for example, scientists saw an association between infection with the virus HIV and the disease, though many health people also carried HIV. There was a plausible mechanism, in that the virus was causing the illness and that a long latency period could explain why so many healthy people were carrying HIV. Eventually they came up with a detailed mechanism for how the virus spreads, infects cells, and causes the symptoms of the disease.

There were also plausible mechanisms behind the early contentions that smoking causes cancer and adding carbon dioxide to the atmosphere changes the climate. Now there are detailed mechanisms.

The author of the paper announcing the diet soda study points to two possible mechanisms. The more plausible was that diet soda makes people fat by disrupting their intestinal microbes. A single, preliminary study showed that indeed diet soda changes our internal zoos, but the weakness in the argument is that almost everything we ingest affects the population of gut microbes. And less plausibly, she suggests that it’s the acid in the diet sodas that causes fat gain, though there’s good evidence that our stomach acids are much stronger than those in anything we eat or drink. Therefore the acidity of our food has no influence on the acidity in our intestines, where those all-important bugs reside.

And there are plausible mechanisms by which the same data might be explained without diet soda causing people to gain weight. It might be, for example, that a tendency toward compulsive eating or sugar cravings causes both the weight gain and the tendency to reach for diet sodas instead of water.

In his piece for science-based medicine, Dr. Crislip made a critical point, which was echoed by several scientists interviewed for my story on cell phones and cancer:

“There is a difference between what is not yet known but possible…..and what cannot exist without a radical rewrite of all of science.”

As an example, he writes that scientists were surprised to find that bacteria called H pylori caused gastric ulcers. But this finding didn’t violate the laws of biology. In contrast, homeopathic medicine violates our most basic understanding of chemistry and physics. (Homeopathy is based on the idea that you can dilute a substance until there are zero molecules left, and it will work based on the “memory” of what it had contained.)

In evaluating claims, it also helps to pay attention to whether scientists are guessing at a plausible mechanism or explaining what they’ve learned from experiments. Doctors can claim that an antidepressant works by rebalancing the brain chemical serotonin, and that’s plausible, but it’s just a guess unless the mechanism has been tested with actual experiments - which leads to another one of Hill’s criteria

Experiment: There are studies and there are experiments and these are not necessarily the same thing. Scientists can follow large groups of people and find connections between eating certain foods and getting cancer, but until they do experiments, they don’t know if the food causes the cancer. It may be that the food is consumed by poorer people with lack of basic health care and it’s the latter that’s a more likely cancer cause.

Science – the process – has a great track record when it comes to revealing the workings of the body, the planet, and the universe. But scientists are like people in any other profession – some are better at their work than others. Even the most brilliant ones can become so emotionally tied to their results that they fail to think critically. As physicist Richard Feynman once said about science, “The first principle is that you must not fool yourself….and you are the easiest person to fool.”