Science news articles abound, everything from the results of carefully designed peer-reviewed research studies to fear-based rumors and anti-science biased conspiracy theories. How are we to discern which are reliable, which are hype based on misinterpretation, flawed studies, and the like, and which are clickbait nonsense?
The first thing I do is look at the source. Mediabiasfactcheck and other sites provide information as to the right-left biases and factual accuracy of a given source, although not of a particular story. Science Based Medicine is also helpful. I’ve been known to search under “Is Dr. So-and-So a quack?” and get useful answers.
I also check my own reactions: Is this too good to be true? Is it at odds with what I understand about science (my academic background is biology and health sciences)? Have I seen an article in a trusted source (such as the newsletter from Center for Science in the Public Interest) debunking this or similar claims? I’ve been also known to check with friends with special expertise in the field.
The Conversation offers some guidelines on assessing the quackery scale of science new stories. Their suggestions:
1. Has the story undergone peer review?
Scientists rely on journal papers to share their scientific results. They let the world see what research has been done, and how.
Once researchers are confident of their results, they write up a manuscript and send it to a journal. Editors forward the submitted manuscripts to at least two external referees who have expertise in the topic. These reviewers can suggest the manuscript be rejected, published as is, or sent back to the scientists for more experiments. That process is called “peer review.”
Research published in peer-reviewed journals has undergone rigorous quality control by experts. Each year, about 2,800 peer-reviewed journals publish roughly 1.8 million scientific papers. The body of scientific knowledge is constantly evolving and updating, but you can trust that the science these journals describe is sound. Retraction policies help correct the record if mistakes are discovered post-publication.
How long has this work been on the preprint server? If it’s been months and it hasn’t yet been published in the peer-reviewed literature, be very skeptical. Are the scientists who submitted the preprint from a reputable institution? During the COVID-19 crisis, with researchers scrambling to understand a dangerous new virus and rushing to develop lifesaving treatments, preprint servers have been littered with immature and unproven science. Fastidious research standards have been sacrificed for speed.
A last warning: Be on the alert for research published in what are called predatory journals. They don’t peer-review manuscripts, and they charge authors a fee to publish. Papers from any of the thousands of known predatory journals should be treated with strong skepticism.
2. Be aware of your own biases.
Beware of biases in your own thinking that might predispose you to fall for a particular piece of fake science news.
People give their own memories and experiences more credence than they deserve, making it hard to accept new ideas and theories. Psychologists call this quirk the availability bias. It’s a useful built-in shortcut when you need to make quick decisions and don’t have time to critically analyze lots of data, but it messes with your fact-checking skills.
A confirmation bias can be at work as well. People tend to give credence to news that fits their existing beliefs. This tendency helps climate change denialists and anti-vaccine advocates believe in their causes in spite of the scientific consensus against them.
3. Correlation is not causation!
Just because you can see a relationship between two things doesn’t necessarily mean that one causes the other.
Even if surveys find that people who live longer drink more red wine, it doesn’t mean a daily glug will extend your life span. It could just be that red-wine drinkers are wealthier and have better health care, for instance. Look out for this error in nutrition news.
4. How were the experiments designed?
The gold standard is a double blind (neither the researchers nor the subjects knew which group got the active ingredient), placebo-controlled (half of the subjects didn’t), with a large enough test population. And was it tested on rats or mice — or humans?
5. Do the “experts” have an ax to grind?
Beware of medical products and procedures that sound too good to be true. Be skeptical of testimonials. Think about the key players’ motivations and who stands to make a buck.