“The three most dangerous words in medicine are “In my experience.” Dr. Mark Crislip, MD
Personal anecdotes and testimonials are one of the most common validations we encounter, particularly when somebody wants to provide proof that something “works”. This is often the case for painful conditions, rehabilitation, training, weight loss, and diet. We have to actively remind ourselves that anecdotes are only a story we tell ourselves or to other people.
In this article, I will explain why you should be very careful not to draw firm (or even any) conclusions based solely on anecdotes, especially your own.
The first red flag is that anecdotes are not valid; are because advertisers and marketers will often rely heavily upon this type of proof. Essentially, they can make it say whatever they want it to express to fit their marketing methods.
“If you are basing your claims on anecdotal experience, then any treatment will seem to work for anything and everything”
Dr. Steven Novella, Assistant Professor of Neurology at the Yale School of Medicine
Anecdotes are also an often-used tactic in debates (even in academic debates); when confronted with proof that disproves a belief, it is effortless to fall back to personal experiences and anecdotes. Claiming “but I have seen it work” or “it worked for me”. However, most people do not realize that this does not strengthen their argument. The possible reason for this is the high value and validity most people place on their own experiences (but not on others’ experiences) and the erroneous way most people think.
What I write about here applies to all people, also myself. We can all easily fall prey to a strong emotional anecdote that confirms our beliefs, especially if we are not actively thinking and wondering if there could be any other explanation. A good exercise is to ask yourself, could there be another explanation for what happened?
A basic rule of critical thinking is that the more extraordinary a claim is, the stronger the evidence is needed to show the claim to have a high possibility of being true. If there is a risk that it is too good to be true, then it often is!
Anecdotes are not evidence. The primary weakness of anecdotes as evidence is that they are uncontrolled; technically, they are non-systematic observations. There is a considerable risk of subconscious data mining, and they are subject to confirmation biases, memory effects, and multiple other cognitive biases. There also is a high risk of confounding variables. Therefore we cannot make any reliable assumptions or show causation from anecdotes.
“The problem with anecdotal evidence, is that there are so many other explanations for what is being claimed” Chris French, Professor of Psychology at Goldsmiths College, University of London
The second red flag is that non-specialists often rely on anecdotes and testimonies. As I have stated before, marketers will often depend heavily on this type of evidence because they can make it say whatever they want it to say. Unfortunately, our brain has a lot of flaws, these including flaws in our memory, our perception, and our thinking. So I’m starting to think that we all are not so reliable.
Some of these flaws in our logic and our thinking are confirmation bias, gambling fallacy, availability heuristic, escalation of commitment heuristic, effort heuristic, fundamental attribution error, the anchoring effect, the “toupée” fallacy, post hoc ergo propter hoc fallacy, attentional bias, congruence bias, bandwagon effect, wishful thinking bias, barnum effect, choice-supportive biases, negativity bias, observation selection bias, observer-expectancy effect, inattentional blindness, change blindness, source amnesia, base rate fallacy to name a few.
Our world has become increasingly more complex than it was only a few hundred years ago. The use of anecdotes to explain this complexity no longer works for us to understand complex phenomena in the world around us. But unfortunately for us, anecdotal stories still hold an undue emotional power over us. By blindly believing in our own experiences, we effectively blindfold ourselves towards looking for other explanations.
“The thing that I think separates skeptics from non-skeptics, more than anything else, is a thorough knowledge of the mechanisms of self-deception, one of the key things of being a skeptic is the humility to know that we can’t rely on anything that we think, that we know” Dr. Steven Novella, Assistant Professor of Neurology at the Yale School of Medicine
Another problem is that we often only look for stories and information that support what we believe, and we choose to avoid, forget or ignore information that is not consistent with our view of the world (1). Yet another error is that the more we have invested in a particular thought or belief, the harder we have to see it as erroneous (2).
An excellent example of how our view of reality gets distorted is when people shall do the simple task of accounting for what food they have eaten during the day. These representations can differ by as much as 48% from reality and lead to significant errors when people tell their stories about how they ate for a certain period (3). People also often miss events if they are engaged in another task (4); this phenomenon is known as “inattentional blindness.” even experts in a field are subject to this bias (5).
So, because of all of these flaws and biases, it is apparent why we can’t show something as complex as to cause and effect from mere anecdotes or make any reliable assumptions based upon them. The reliability and validity are way too low for this, so anecdotal stories are not valid as evidence. Now, science is far from perfect, and it has its own flaws, but it is still a whole lot better than blindly trusting our own subjective experience.
Even if we could be sure of what we experience (we can’t) and the effect we experience (that it works), is accurate. We still can’t say what works, only that something “works”. Because anecdotes are still uncontrolled observations, and as such, there is a considerable risk of confounding variables. Said another way, there is a significant risk that there is a hidden factor that influences the effect that we did see, or even that it is the hidden factor that caused the effect. What we see could also just be a correlation, and correlation does not imply causation, only correlation, with the possibility of causation.
“When you start down the road where belief in magic replaces evidence and science, you end up in a place where you don’t want to be.” Michael Specter
The primary use of anecdotes is to entertain and tell people’s own experiences and should not be taken as good and sound evidence for anything. Anecdotal evidence still has some value; there’s nothing wrong with using anecdotal evidence as long as one understands anecdotes are not evidence of anything and that you might just be fooling yourself. People often also mistake thinking that the more anecdotes, the more solid proof there is for the treatment or method in question.
”The plural of anecdote is not ‘data”. Frank Kotsonis
Anecdotal stories have no place in the hierarchy of validity (6, 7), but case studies can have some merit. Often practitioners are tempted to base clinical convictions on personal experience (8), but this is not recommended because of the high risk of errors. Documented anecdotes are sometimes used in science in the form of case studies. However, case studies should not be used to show what to do. But they can have great value in illuminating an area that might need research and can be very useful in forming a new testable hypothesis.
An excellent summary of (9) about the flaws there is with using anecdotal evidence is found in “A Practical Study of Argument” by Trudy Govier:
“Arguments in which there is an attempt to base an inductive generalization on a story or anecdote are often said to be anecdotal arguments. In many cases, the stories involved are taken from the personal experience of the arguer. Since no personal experience can provide a broad enough sample of the world to support a generalization, the reasoning is hasty and fallacious. While the anecdotes told may be interesting and vivid, and may serve valuable roles in communication, they cannot provide enough evidence for a strong inductive argument.”
So the recap advertisers and marketers will often rely heavily on personal anecdotes; they are uncontrolled non-systematic observations, and we can’t make any reliable assumptions from anecdotes. We can’t show causation from anecdotes.
This is why healthcare in the 21st (twenty-first) century should be based upon quality research and scientific thinking to safeguard us against our flawed thinking and potentially biased observations and experiences.
I would like to thank exercise physiologist Anders Nedergaard, PhD, and physiotherapist Jacob Gudiol for the inspiration and research which have helped me write this article.
“A wise man proportions his belief to the evidence” David Hume
1. Jonas E. Traut-Mattausch E. Frey D. Greenberg J. The path or the goal? Decision vs. information focus in biased information seeking after preliminary decisions. Journal of Experimental Social Psychology 44 (2008) 1180–1186.
2. Ku G. Before escalation: behavioral and affective forecasting in escalation of commitment. Pers Soc Psychol Bull. 2008 Nov;34(11):1477-91. Epub 2008 Aug 28.
3. Hill RJ. Davies PS. The validity of self-reported energy intake as determined using the doubly labelled water technique. Br J Nutr. 2001 Apr;85(4):415-30.
4. Most SB1, Simons DJ, Scholl BJ, Jimenez R, Clifford E, Chabris CF. How not to be seen: the contribution of similarity and selective ignoring to sustained inattentional blindness. Psychol Sci. 2001 Jan;12(1):9-17.
5. Drew T1, Võ ML, Wolfe JM. The invisible gorilla strikes again: sustained inattentional blindness in expert observers. Psychol Sci. 2013 Sep;24(9):1848-53. Epub 2013 Jul 17.
6. Cook DJ, Guyatt GH, Laupacis A, Sackett DL, Goldberg RJ. Clinical recommendations using levels of evidence for antithrombotic agents. Chest 1995 Oct; 108(4 Suppl):227S-230S.
7. Wright JG, Swiontkowski MF, Heckman JD. Introducing levels of evidence to the journal. J Bone Joint Surg Am. 2003 Jan;85-A(1):1-3.
8. Steve E Hartman. Why do ineffective treatments seem helpful? A brief review. Chiropr Osteopat. 2009; 17: 10. Published online Oct 12, 2009.
9. Trudy Govier. A Practical Study of Argument, 7 edition. Cengage Learning; 7 edition (January 1, 2013)