Inconsistent Reporting of Methodologies Makes Most Radiation Biology Studies Impossible to Replicate

UMSOM Researchers Find the Vast Majority of Radiation Oncology Studies Lack Proper Reporting of Methodology.

Nearly 80 percent of radiation oncology studies funded by the National Institutes of Health involve investigating the effects that radiation has on tumor cells and healthy tissue in pre-clinical settings, such as experiments done in cell cultures or mice. A majority of these radiation biology studies, however, have serious flaws in how their irradiation methodology is described, which makes them very difficult to replicate, according to a new finding from the University of Maryland School of Medicine (UMSOM).

Important details in the irradiation protocol and the experimental setup are routinely not included in most of these journal articles, which could lead to dose variations or other errors when other researchers try to repeat the experiments in their own laboratories.

“Glaring omissions or errors in the methodology sections of radiation biology journal articles make the studies very difficult to reproduce, interpret, and compare with other research,” said study principal investigator Yannick Poirier, PhD, Assistant Professor in the Department of Radiation Oncology at UMSOM.

In the new study published this month in the International Journal of Radiation Oncology Biology and Physics, Dr. Poirier and his colleagues from UMSOM and the University of Washington, Seattle, reviewed 1,758 peer-reviewed studies from 469 journals that were used in pre-clinical studies to evaluate the effects of radiation therapy to treat malignant tumors and other conditions. Specifically, they examined the journal article’s “methods” section that described the radiation protocol used.

The researchers found that the source of radiation used in the study (such as low vs high energy X-rays or gamma rays) was unclear or ambiguous in nearly 14 percent of the studies they reviewed. They also found that only 1 percent of studies listed the protocol the researchers used to calibrate the machines, and only 16 percent named the equipment used to measure the absorbed radiation dose.

Several publications were found to contain outright errors where a quantity of radiation dose was misreported. In a few cases, researchers found descriptions of “unachievable” experiments including irradiation produced by a linear accelerator at energy levels that would have been impossible for the named device to produce.

The authors graded the level of physics reporting necessary to successfully reproduce the experiment on a scale of 1 (worst) to 10 (best) and found that only 3 percent of the journal articles would receive a score of 8 or above. “Disturbingly, articles published in higher impact journals -- and, consequently, those that are higher cited -- scored the poorest. This means that these studies with poor-quality physics are being propagated and are amplifying the reproducibility problem,” said study co-author Amit Sawant, PhD, Associate Professor and Chief of the Division of Physics in the Department of Radiation Oncology at UMSOM.

The UMSOM researchers stipulate that their findings are limited to the physics and irradiation aspects of these radiation biology studies. The quality of the science, hypotheses, or non-physics aspects of the experimental design were not reviewed by the UMSOM researchers.

“This inability to reproduce scientific findings could profoundly impact the translation of preclinical research results into clinical practice,” said E. Albert Reece, MD, PhD, MBA, Executive Vice President for Medical Affairs, UM Baltimore, and the John Z. and Akiko K. Bowers Distinguished Professor and Dean, University of Maryland School of Medicine. “The UMSOM researchers highlight a crucial issue that should be addressed by the radiation oncology research community.”

The failure to report or reproduce radiation absorbed dose in radiation biology journal articles is likely due to a lack of consultation between the radiation biologists performing the study and radiation physicists who have the expertise to design, document and validate the radiation delivery protocol, according to the UMSOM researchers.

“We are raising awareness of this issue because the problem can be fixed by involving more properly trained physicists in these studies,” said Dr. Poirier. “That is because radiation delivery follows well-understood radiation physics principles that lend themselves to high accuracy, precision, and reproducibility.”

Efforts are already underway to address the problem: NIH’s National Cancer Institute and National Institute on Allergy and Infectious Diseases recently formed a program to standardize and monitor the radiation dosimetry delivery among the entirety of their preclinical research program funded by the radiation countermeasures program. The American Association of Physicists also established a task force last year to create guidelines for accurate dosimetry in radiobiology experiments.

About the University of Maryland School of Medicine

Now in its third century, the University of Maryland School of Medicine was chartered in 1807 as the first public medical school in the United States. It continues today as one of the fastest growing, top-tier biomedical research enterprises in the world -- with 43 academic departments, centers, institutes, and programs; and a faculty of more than 3,000 physicians, scientists, and allied health professionals, including members of the National Academy of Medicine and the National Academy of Sciences, and a distinguished recipient of the Albert E. Lasker Award in Medical Research.  With an operating budget of more than $1 billion, the School of Medicine works closely in partnership with the University of Maryland Medical Center and Medical System to provide research-intensive, academic and clinically based care for more than 1.2 million patients each year. The School has over 2,500 students, residents, and fellows, and more than $530 million in extramural funding, with most of its academic departments highly ranked among all medical schools in the nation in research funding. As one of the seven professional schools that make up the University of Maryland, Baltimore campus, the School of Medicine has a total workforce of nearly 7,000 individuals. The combined School and Medical System (“University of Maryland Medicine”) has an annual budget of nearly $6 billion and an economic impact more than $15 billion on the state and local community. The School of Medicine faculty, which ranks as the 8^th highest among public medical schools in research productivity, is an innovator in translational medicine, with 600 active patents and 24 start-up companies. The School works locally, nationally, and globally, with research and treatment facilities in 36 countries around the world. Visit medschool.umaryland.edu