Richard Harris titled his book Rigor Mortis, referring to the stiffening of the body after death, to convey that biomedical science as it is currently practiced suffers from a lack of rigor. It is a pun he must like, because he employs it very early and very often.
The problem Harris is bemoaning is large and legitimate. Drug trials are incredibly expensive in terms of the time and money spent by the government and researchers—as well as the pain, dashed hopes, and even deaths of the patients enrolled. These drug trials are often based on suggestive findings from basic research done in academic labs, findings like compound X (green tea, vitamin E, whatever) fixes cells or cures animals with disease Y (diabetes, cancer, etc.). If that basic research is flawed, of course, the drug trials will fail.
Harris reports that drug trials do, in fact, often fail. Their failure, he writes, is largely, though not completely, because much of the basic research upon which they are based is enormously flawed.
Reproducibility
Harris, who is an NPR science correspondent, starts off Rigor Mortis by describing the widely documented “reproducibility crisis.” Although problems with reproducibility are most often associated with psychology, the same issues turn out to be endemic throughout much of biomedical science. Although the notion that a researcher’s results should be reproducible in any other lab in the world is sacred, in reality, this notion is rarely tested. And when people started testing it, results did not fare well. Not only did other labs fail to reproduce seminal results that were widely cited in their respective fields; often, the original researchers could not even reproduce their own results.
Some of the reasons lab results can be wrong—or at least irreproducible—are technical. One, which would seem silly if its effects weren’t so wide-ranging and tragic, is that researchers don’t validate their cell type before each experiment. Harris claims that “between 18 and 36 percent of all cell experiments use misidentified cell lines.” This can be because of accidental contamination or an honest mistake when the cell line was first isolated decades ago. But misidentification means that labs that think they’re working with breast cancer cells, for example, may in fact be using melanoma cells. Hence failed drug trials. Other vital reagents, like antibodies, similarly go untested.
Another obstacle to reproducibility is the reliance on animal models. Mice, clearly, are not men; researchers have been able to cure diabetes in mice a couple of hundred times over. Harris writes, “one reason everybody uses mice: everybody else uses mice.” The infrastructure and technology are all in place and therefore convenient, and the alternatives are limited. But the clinical relevance and applicability of these studies is still very variable.
Cells and animals are also both subject to batch effects. These are properties that differ from lab to lab and day to day. Even if they can be identified, batch effects often can’t be controlled. Mice react differently to male and female researchers (males stress them out and alter their hormone levels); they react differently when music is playing. Cells act differently when grown in one brand of Petri dish as opposed to another, and they are responsive to the temperature and humidity levels in a room. This means that samples that were not handled at the same time—be they separated by miles, years, or even hours—cannot always be meaningfully compared to each other.
Genomes and solutions
Genomic studies provide a whole new set of technical issues. Researchers are now combing through bytes upon bytes upon bytes of data every hour searching for tiny but “statistically significant” correlations between genetic sequences and clinical outcomes. Because these studies are trying to be exquisitely sensitive to finding small effects, they also end up being exquisitely vulnerable to red herrings. Many biologists are not trained in statistics and do not properly apply statistical methods to their data sets. With big data come big mistakes and big problems.
Other problems plaguing lab research are cultural. One issue is that mistakes are built into the scientific method: researchers base their hypotheses upon the best available evidence and then revamp their hypotheses when new evidence becomes available. This is how the system is supposed to work, and we need to be aware of how provisional some results may be.
Complicating matters is that scientists are rewarded with funding and with tenure for being first, not for being right. If their published results are later disproved, well, getting their work into that high-profile publication was probably still worth it. This is not because scientists are especially greedy or ambitious or egotistical (no more so than any other humans), but funding and tenure are very, very, very hard to come by.
What next
All is not doom and gloom, though; Harris offers solutions. Things like validating cell lines and antibodies are pretty straightforward; other fixes will be more complicated. Currently, research labs are often like little fiefdoms, with each investigator passing on techniques learned at the feet of his or her mentor. Biomedical research has no checklist like Atul Gawande promoted in medicine or standards like the good institutional practice that exist in the pharmaceuticals industry. These can be mandated and implemented, and a movement to do just that is already in the works.
There is also a movement for basic scientists to register their hypotheses in advance. This means they won’t be able to move the goalposts after the data come in and claim to have found something they were never actually looking for in the first place. This registration has been required of clinical trials since 1997 (although it has not been implemented as extensively as it could be).
Rigor Mortis is rife with examples of things that go awry in medical studies, how they happen, and how they can be avoided and fixed. For the most part, academic biomedical scientists are not evil, malicious, or liars at heart. Harris knows that they are predominantly seekers of Truth who cannot follow their curiosity wherever it leads them because they have to make a living, like everyone else. They are stuck in a system that only funds conservative research and rewards Important, Ground-Breaking Results™.
Presumably, these scientists are part of the target audience of Rigor Mortis. But as the author himself points out, they are probably too busy writing grant applications to get to the book and hear its potentially life-saving message.
reader comments
68