Assisting teachers and families in decision-making regarding pseudoscientific interventions
With the advent of the internet, an increasing number of questionable and pseudoscientific interventions are available to teachers and families. While some of these interventions are little more than moneymaking scams, the majority appear to be offered by sincere but misguided proponents. So what sources of information should teachers and families rely upon and what red flags might signal that additional caution is needed in evaluating claims?
By Mark Carter
Where should teachers and families look for information about interventions or programs they are considering? An obvious starting point is to look for an ‘expert’, typically someone with a doctorate relevant to the area. For example, if you are seeking veterinary advice, you might approach a close relative of mine, Dashiel Carter. Dashiel has a Doctor of Science from the prestigious Thunderwood College in California, USA. A copy of his testamur is presented in evidence in Figure 1.
The title of Dashiel’s thesis was ‘A qualitative analysis of pareidolia in veterinary homeopathic treatment of travel sickness in Canis Lupus Familiaris: You can lead a dog to water but does it have magical healing properties?’ There is just one small problem, which should become obvious from the photograph of Dashiel in Figure 2. Yes, Dashiel is a 12-year-old whippet. And before you ask, no, my doctorate is not from Thunderwood College.
Even if your expert doesn’t turn out to be a dog (literally), relying on the advice of a single professional can be problematic. There are many examples of individuals with relevant expertise and qualifications who hold views that fall well outside the scientific consensus – for example, paediatricians who are vaccine denialists, scientists who are HIV denialists and educators who deny the accumulated scientific evidence on early reading acquisition or systematic instruction.
So, how should professionals and families judge the reliability of ‘expert’ advice? Probably the best (but far from perfect) option is to look for a consensus of experts. This can include consensus and position statements from major professional bodies and associations. For example, facilitated communication (recently renamed the rapid prompting method) is a pseudoscientific intervention that is used with individuals with disability, particularly autism. Based on accumulated evidence from scientific research and reviews, position statements refuting facilitated communication have been issued by numerous professional organisations including the American Speech-Language-Hearing Association (2018), the International Society for Augmentative and Alternative Communication (2014) and the American Academy of Pediatrics (1998). Such position statements provide a good starting point but, obviously, depend on whether the association or organisation bases its position on scientific evidence or ideology. The next best option is the individual expert with relevant background, such as a researcher who is actively publishing in the area. More caution should be exercised with individual experts, who are more likely to be biased or plain wrong on a given issue. The final option is the individual, often self-proclaimed, ‘expert’ without relevant background or qualifications, whose advice is likely to be unreliable. For those who think this is a good option, Dr Dashiel is always available for consultation, at a fee.
What about the internet?
The internet has brought vast quantities of information into people’s homes – and phones. However, this information is usually unfiltered and can be of poor quality. When people say they “researched” an area, they often mean they did little more than a Google search and selected information that conforms with their pre-existing worldview. You don’t even need to be that selective as search engines will typically feed you information that conforms with your pre-existing views, based on your prior search history. There are certainly reliable sources of information on the internet but it can be difficult to sort the wheat from the chaff.
But I saw it with my own eyes: anecdotes and testimonials
Perhaps the most convincing sources of information that we encounter are anecdotes and testimonials including advice from friends, testimonials on the internet or the media, and product endorsements. Humans seem inherently attracted to stories and we often give undue weight to personal anecdotes over dry sterile evidence. Unfortunately, when individuals state, “I saw it with my own eyes”, they fail to realise that we often see what we want and expect, and are easily deceived. In addition, memories of events are often faulty and are reconstructed over time. In addition, testimonials may reflect rare apparent successes; proponents are unlikely to showcase critical or negative testimonials. Of course, when commercial interests are involved, testimonials may well simply be fabrications. In addition, perception and memory tend to be selective and we attend to hits that are consistent with our views and tend to forget the misses that are not. Anecdotes and testimonials are inherently vulnerable to numerous cognitive biases that affect humans and may result in less than optimal decision-making. Testimonials can be considered illustrative but do not constitute evidence.
Scientific research evidence
Scientific studies provide our most reliable way of establishing what works (eventually) as this research attempts to control the biases that affect all of our judgements. I say ‘eventually’ because it is often a messy process. By its nature, research is tentative and self-correcting. Research can vary in quality and it often takes years or decades for sufficient high-quality research to surface to allow a clear picture to emerge. Many initial findings (often based on low-quality preliminary studies) turn out to be incorrect and the process of reaching a consensus relies on accumulation of evidence from many studies and researchers.
The process of interpreting research can be difficult, time-consuming and nuanced, so where do you start? A good place is to look for reviews of interventions in peer refereed journals. Ideally, you want to look for a number of reviews to determine if the authors converge on a conclusion about the intervention. Importantly, when interpreting reviews, make sure you look at the authors’ comments on the quality of the research, as this will constrain the strength of any conclusions they might reach. In the absence of reviews, you can look for individual studies. Relying on the results of a single study is always problematic so, ideally, you will be looking for several studies that converge on similar findings. It is often worthwhile, also, looking to see whether others have critiqued or criticised the existing studies. Poor quality research on controversial interventions will often draw a response from the scientific community.
A word of caution is warranted. Academic publishing has become big business with some publishers generating profits that would not be embarrassing for a medium-sized pharmaceutical company. Unfortunately, this has led to large numbers of ‘predatory open access journals’ being established to cash in on the boom. In 2015, it was estimated that more than 10,000 of these journals existed and growth since then is likely to have been exponential. These journals typically publish manuscripts for money, have little or no quality control and submitted articles do not go through normal scientific vetting procedures. Not capitalism at its finest. A search for lists of “predatory open access journals” will provide a number of sources that you can use to assist in assessing the likely reliability of a journal.
Do the proponents make extravagant claims such as cures and miracles? Do they claim their intervention addresses many different types of problems? If so, caution is warranted.
Does the intervention make sense in terms of existing knowledge? For example, reading problems are not typically due to visual-perceptual issues so a program focusing on visual perception should be treated with caution.
Are anecdotes and testimonials presented in the absence of research evidence? Anecdotes and testimonials are advertising tools and may be useful to illustrate an intervention. As such they are legitimate marketing tools. They should, however, not be considered alternatives to research evidence and provide no reliable indication of whether an intervention or program is effective. Best advice would be to look past the testimonials and go direct to the research evidence.
Has any research been published in peer-refereed journals and, if so, has it been criticised or independently replicated? While much research published in peer-refereed journals is of poor quality, these journals do at least provide a limited level of initial quality control. If research has been published, has it been subject to criticism or rebuttal? Ideally, research should also be independently replicated by those other than the program developers.
Do the proponents have a commercial interest? To be clear, I am not a communist and I am happy for individuals to profit from their endeavours. However, when commercial interests are involved it is prudent to exercise an extra degree of caution in interpreting claims. In these cases, it is particularly important to carefully look at the research evidence supporting the claims.
Do the proponents claim a conspiracy by the scientific establishment? A common feature of pseudoscience is a claim that intervention is being withheld or covered up by the scientific establishment. Such claims are non-falsifiable – if you argue an intervention is ineffective, you are clearly part of the conspiracy!
Does it claim to be ‘natural’? The claim that intervention is ‘natural’ is a marketing exercise and tells us very little about the nature of the intervention. Both smallpox and Ebola are ‘natural’ but you probably don’t want to sprinkle them on your breakfast cereal. A claim that something is natural tells you nothing about its safety or efficacy.
In summary, we all make less than optimal decisions due to many inherent cognitive biases. Rigorous research offers us the best option to compensate for these flaws and gain valid information on the effectiveness of interventions and programs. In making decisions, teachers and families would be well advised to keep a watch for red flags that may signal questionable or pseudoscientific interventions or programs.
This article was published in the October 2019 edition of Nomanis.
Mark Carter has contributed to the field of special education for over 30 years. He is currently an Associate Professor in Special Education at Macquarie University. Mark has a range of research interests including educational issues in autism spectrum disorder, evidence-based practice in special education and controversial practices. He has over 100 refereed publications and book chapters.