The results here are much less obvious. The extensive supplemental material provided by the replication team helpfully distinguishes between “reproducibility” (whether the results of an experiment are the same if you run them again with the same data and approach?) and “regenerative ability(Can a new, overlapping experiment with new data reliably yield similar results?).
The COS team tried to make it clear how messy all this is. If a test is not reproducible, that does not mean it is inexplicable. It could be a problem with the copy, not the original work. Conversely, an experiment that someone can perfectly reproduce or reproduce is not necessarily true, and it is not necessarily useful or novel.
But the truth is, 100% pure copying is not really possible. Even with the same cell line or the same genetically edited mouse line, different people perform the experiments differently. Maybe the ones that the cloning team doesn’t have the documentation to complete will do better. It is possible that “high impact” articles from the most prestigious journals are bolder work, take risks, and are less likely to be repeated.
Cancer biology has high stakes. After all, it is said to lead to life-saving drugs. The non-replication work on Errington’s group is unlikely to result in any dangerous drugs or harm any patients, because Phase 2 and Phase 3 trials tend to screen for those bad seeds. According to the Biotechnology Industry Organization, only 30% of drug candidates pass Phase 2 testing and only 58% pass Phase 3. (Good for determining safety and efficacy, not good for blowing all the money out of research. and increase drug costs.) But drug researchers admit, to put it mildly, that most approved drugs don’t work well at all — especially cancer drug.
Overall, the science is clearly working. So why is it so difficult to reproduce an experiment? “One answer is: Science is hard,” says Errington. “That’s why we fund research and invest billions of dollars just to make sure that cancer research can have an impact on people’s lives. Which it does. ”
The point of results that aren’t as big as the cancer project is to distinguish between what’s good for science internally and what’s good for science when it comes to ordinary people. “There are two orthogonal concepts here. One is transparency and the other is validity,” said Shirley Wang, an epidemiologist at Brigham and Women’s Hospital. She is the driver of the Reproducible Evidence Initiative: Practice to Enhance and Achieve Transparency (“Iterative”), which has performed replicated work on over 150 studies using records. electronic health records as their data. (Wang’s Repeat article has yet to be published.) “I think the point is that we want the convergence of the two,” she said. “You can’t tell if it’s good quality science unless you can be clear about the methods and reproducibility. But even if you can, that doesn’t mean it’s a good science.”
Therefore, it is not a matter of criticizing specific results. It’s to make the science more transparent, which in turn will make the results more replicable, easier to understand, even transferable to the clinic. Currently, academic researchers have no incentive to publish work that other researchers can copy. Promotion is just for publishing. “A measure of success in academic research is getting an article published in a leading journal and the number of citations that paper has,” says Begley. “For industry, the measure of success is the drug on the market that works and helps the patient. So we at Amgen couldn’t invest in a program that we knew from the start was not really legit.”
