Absence of Evidence is Not Evidence of Absence

Background

My clients and followers often ask me about the effectiveness of a certain technique, device, or supplement. In cases, my response will be something along the lines of, "We don't know." The questioner will then look at me bewildered and ask how that can be given (insert fitness professional here) said it was beneficial. Surely it is or isn't? It all comes down to what information we actually know. Let's delve a little deeper.

The Common Responses

Let's first begin by noting that the following discussion refers to fitness and health professionals that mean well when they give their answers. There are unfortunately those out there who do not mean well and are not doing their due diligence when they spread information. This is an entirely different issue, beyond the scope of this article. This being noted, when asked if a health or fitness product/technique is effective, common responses by well-meaning professionals are as follows:

• Yes (X affects Y).
• No (X doesn't affect Y).

Although we often think of "yes" and "no" as clear, straightforward answers to a question, when were are talking about an effect in science, there are very specific meanings to these words. Properly using these words as an answer means the following:

• Yes: There are enough studies of high enough quality to say, at this time, that there is a significant effect.
• No: There are enough studies of high enough quality to say at this time that there is no significant effect.

Notice the key parts that define both a "yes" and a "no." There needs to be enough studies, the studies need to be of a high quality, and they need to point towards a "yes" or a "no." in a significant manner (i.e., backed by statistical analysis). If there are not enough studies, or the studies we have are of poor quality, or lack of significance, then one should steer clear of a simple "yes" or "no" answer and be more specific in terms of what evidence we have.

Embedded in the above definitions is, unavoidably, a bit of subjectivity. The subjectivity comes in determining how many studies is "enough." How many high-quality, significant studies do we need to have in order to feel confident that there an effect (or no effect). Unfortunately, there isn't a hard and fast rule here. The number needed for confidence will often be different if you ask different researchers in the field of study. In addition, if all published studies point to the same conclusion, researchers may be more confident in giving a "yes" or "no" with fewer studies, in comparison to a situation where some studies point to different conclusions (researchers may then want to see more studies come out before saying "yes" or "no"). By bringing this up, my intent was not to get us lost in the weeds, but to make you aware of a consideration which we will touch more on later.

The Less Common Responses

Although communication can be improved with the definitions I listed above for "yes" and "no," they don't cover all bases. As a theoretical example, what if someone asked whether genetically modified parsnips negatively affect lifespan? Most fitness professionals will answer, “There is no evidence indicating genetically modified parsnips will affect lifespan.” Although this may be a technically accurate answer, it is a bit misleading. Many people hear this phrasing and take it to mean that no genetically modified parsnip studies have shown an effect on lifespan. While this could be the case, it also could be the case that no studies were conducted on the topic whatsoever. If this is the case, a better response would be to say:

• We do not know: This question has not be studied as of yet.

So, again, to be rigorous, if no studies have been conducted, we should not say that no effect exists. As an aside, if there were studies of other genetically modified vegetables on lifespan you could make a statement that the effect they have on lifespan is likely to also be found for modified parsnips. It's still a prediction (hypothesis), but it is a reasonable application of evidence.

What about a situation where we don't have enough studies? For these situations it is best to to say:

• It's unclear: We have limited high-quality evidence at this point. The evidence may suggest an effect or no effect in its current state.

In these types of situations, if the limited high-quality evidence points in a certain direction, you can make a note of that, but it's still important to lead, up front, with the effect being unclear at the present time. 

Absence of Evidence is Not Evidence of Absence

Although we just highlighted the importance of saying when we don't have evidence for an effect or phenomena, it's important to remember that a great many things were thought to be ineffectual or non-existent for extremely long periods of time. For example:

1. Genetics: We knew virtually nothing about genetic inheritance before Gregor Mendel started breeding different types of peas in the 1850s.
2. Neurology: It was assumed for years that the neural degeneration in mad cow disease must be caused by some known, but yet to be pinpointed, effect like a hormonal imbalance, aberrant cellular activity, or genetic abnormality. However, Stanley Prusiner had an idea and conducted some experiments showing that misfolded proteins (which he named prions) actually induce the disease, garnering him the Nobel Prize in 1994.
3. Immunology: Similarly the state of the Immunological field was for years tied to the roles of various cell types in regulating immunity (B cells, T cells, macrophages, etc.), until Ralph Steinman came along hypothesizing that there was a missing link. His studies eventually revealed the presence of dendritic cells, a missing link in the body’s ability to combat pathogens. Steinman won the Nobel Prize for this work in 2011.

(Above: Computer-generated rendering of a dendritic cell.)

Before Mendel, Prusiner and Steinman conducted their studies, the world assumed certain things didn’t exist or were not relevant. After the right studies were conducted, our understanding of the natural world was changed substantially. So, remember, just because something has not been investigated, does not mean there is nothing there to be seen. In other words, an absence of evidence doesn't necessarily mean there is evidence of absence.

Why Choosing The Right Answer is Difficult

Why do so many fitness professionals have such a hard time choosing the right answer to questions? The reason is multi-factorial. To come up with the correct answer they need to 1) be aware of all published studies on the topic, 2) have read all these studies and assessed them for quality and significance, 3) have determined that there are enough of these studies pointing to an answer that is different from "we do not know," and 4) understand the difference between the four best practice responses discussed above. Even for someone who has professional training in research this is a daunting, often full-time, endeavor. When you then consider that most well-meaning fitness professionals do not have the training or ability to effectively execute on 1-4, it becomes obvious why erroneous answers may be given.

Final Thoughts

Experts and professionals need to be careful when they talk about what is known and unknown. In this article I made the case to use four best practice general responses to a question of effect/effectiveness (instead of just a yes or no response). These responses are:

• Yes.
• No.
• We do not know.
• It's unclear.

When it comes to making informed decisions about fitness and health, the general public can take the advice or insight of a knowledgeable individual, delve into the relevant published scientific studies to find a consensus themselves, or employ some combination of the two. If you do decide to follow a fitness professional, make sure they understand the differences between the above-mentioned four responses. Professionals that do understand the difference will generally highlight the limitations of individual studies and discuss the quality and quantity of evidence that exists in support of a phenomenon.

 

To receive coaching oversight from Science for Fitness, book a free consultation HERE.