By Sue Rochman
From Mice to Men
The scientific method can help you decide if the latest cancer news is really news
By Sue Rochman
Ready for Prime Time?
Typically, research studies make headline news when they are published in a medical journal or presented at a scientific meeting. But, of course, not all the studies published are from phase III randomized trials. Most describe early research. Furthermore, scientists say, not all of the research touted by the media as a major breakthrough is truly ready for prime time.
This is especially true of the research presented at scientific meetings, some researchers say. “The purpose of scientific conferences is to present emerging findings,” says Weigold. “Researchers understand that these are works-in-progress and not the same as published results, which have been peer-reviewed” by other researchers who are knowledgeable about the subject. Yet, quite often, meeting organizers send out press releases promoting these new findings, and the media, eager to spread the word on the newest advances, report them.
But as a study by Schwartz and her colleagues published in 2002 in the Journal of the American Medical Association showed, scientific and media excitement is a poor measure of success. The researchers identified 147 research abstracts, or summaries, that had been presented at scientific meetings and had received extensive media coverage. Three years after the meetings were held, 25 percent of the research reported in the abstracts had never been published. Schwartz and her colleagues also found that the abstracts that had been highlighted by the meeting organizers or had made front-page news were no more likely to go on to be published in a medical or a scientific journal.
Why the research wasn’t published isn’t known. It could be that the final results contradicted the preliminary findings, or that the researchers never submitted their work for publication. It’s also possible that the researchers did submit their work, but that it never made it through the peer-review process that is designed to weed out problematic study designs, data misinterpretation, or unsubstantiated conclusions.
Even if an early study does get published, it doesn’t mean it’s going to prove fruitful. The need for consumers to understand the false hopes that early studies can engender was underscored in a research letter published in 2006 in the Journal of the American Medical Association by two scientists at the University of Toronto. The researchers had identified 76 promising animal studies published between 1980 and 2000. They then looked to see what had happened to the drugs these studies tested. They found that only 28 of the 76 drugs were later tested and found to work in humans. (Fourteen were not found to be effective and 34 remain untested.) Furthermore, only eight of these drugs went on to receive government approval for use in patients—a success rate of about 10 percent.
This doesn’t mean the public shouldn’t hear about or get enthusiastic about early research. The key is to recognize that early research is just a part of the scientific process, and to learn how to tell what kind of promise it holds.