UM School of Medicine Study Finds ChatGPT Helpful for Breast Cancer Screening Advice, With Certain Caveats

Research Will Lead to Potential New Way to Promote Health Literacy to Consumers

As more consumers turn to the newly available ChatGPT for health advice, researchers are eager to see whether the information provided by the artificial intelligence chatbot is reliable and accurate. A new study conducted by researchers at the University of Maryland School of Medicine (UMSOM) indicates that the answers generated provide correct information the vast majority of the time; sometimes, though, the information is inaccurate or even fictitious.

Findings were published today in the journal Radiology.

In February 2023, UMSOM researchers created a set of 25 questions related to advice on getting screened for breast cancer. They submitted each question to ChatGPT three times to see what responses were generated. (The chatbot is known for varying its response each time a question is posed.) Three radiologists fellowship-trained in mammography evaluated the responses; they found that the responses were appropriate for 22 out of the 25 questions. The chatbot did, however, provide one answer based on outdated information. Two other questions had inconsistent responses that varied significantly each time the same question was posed.

“We found ChatGPT answered questions correctly about 88 percent of the time, which is pretty amazing,” said study corresponding author Paul Yi , MD, Assistant Professor of Diagnostic Radiology and Nuclear Medicine at UMSOM and Director of the UM Medical Intelligent Imaging Center (UM2ii). “It also has the added benefit of summarizing information into an easily digestible form for consumers to easily understand.” ChatGPT correctly answered questions about the symptoms of breast cancer, who is at risk, and questions on the cost, age, and frequency recommendations concerning mammograms.

The downside is that it is not as comprehensive in its responses as what a person would normally find on a Google search.  “ChatGPT provided only one set of recommendations on breast cancer screening, issued from the American Cancer Society, but did not mention differing recommendations put out by the Centers for Disease Control and Prevention (CDC) or the US Preventative Services Task Force (USPSTF),” said study lead author Hana Haver, MD, a radiology resident at University of Maryland Medical Center.

In one response deemed by the researchers to be inappropriate, ChatGPT provided an outdated response to planning a mammogram around COVID-19 vaccination. The advice to delay a mammogram for four to six weeks after getting a COVID-19 shot was changed in February 2022, and the CDC endorses the USPSTF guidelines, which don’t recommend waiting. Inconsistent responses were given to questions concerning an individual’s personal risk of getting breast cancer and on where someone could get a mammogram.

Jean Jeudy, MD, Professor of Diagnostic Radiology and Nuclear Medicine and Radiology Vice Chair of Informatics at UMSOM was a co-author on the study. Researchers from the Massachusetts General Hospital and the Johns Hopkins University School of Medicine also participated in this study. The study was funded by the individual institutions.

“We’ve seen in our experience that ChatGPT sometimes makes up fake journal articles or health consortiums to support its claims,” said Dr. Yi. “Consumers should be aware that these are new, unproven technologies, and should still rely on their doctor, rather than ChatGPT, for advice.”

He and his colleagues are now analyzing how ChatGPT fares for lung cancer screening recommendations and identifying ways to improve the recommendations made by ChatGPT to be more accurate and complete – as well as understandable to those without a high level of education.

“We are witnessing an unprecedented revolution in health care where the integration of artificial intelligence and immersive technologies will fundamentally change the way we treat patients," said Mark T. Gladwin, MD, Dean, University of Maryland School of Medicine, Vice President for Medical Affairs, University of Maryland, Baltimore, and the John Z. and Akiko K. Bowers Distinguished Professor. "ChatGPT and other language models are an exciting part of this transformation – providing access to a vast database of medical knowledge and potentially offering personalized advice based on specific symptoms and a patient’s medical history, albeit with certain limitations, as this study points out.” 

The University of Maryland, Baltimore (UMB), University of Maryland, College Park (UMCP) and University of Maryland Medical System (UMMS) recently launched plans for a new Institute for Health Computing that will leverage recent advances in artificial intelligence and computing to create a premier learning health care system that evaluates both de-identified and secure digitized medical health data to diagnose, prevent and treat diseases in patients. It is the strategic initiative of University of Maryland Strategic Partnership: MPowering the State, which brings together the complementary strengths of UMB and UMCP to strengthen Maryland’s innovation economy, promote interdisciplinary research, and create more educational opportunities for students. 

"Academic medicine has a unique opportunity to lead this new era of medicine, with medical schools on the front lines of biomedical research, access to ‘big data,’ clinical care, and education," said Dean Gladwin.  "Our vision is for this to become the ‘East Coast Silicon Valley’ for health computing -- embracing and harnessing the power of clinical analytics and precision medicine to enhance patient care and provide population health services.”

About the University of Maryland School of Medicine

Now in its third century, the University of Maryland School of Medicine was chartered in 1807 as the first public medical school in the United States. It continues today as one of the fastest growing, top-tier biomedical research enterprises in the world -- with 46 academic departments, centers, institutes, and programs, and a faculty of more than 3,000 physicians, scientists, and allied health professionals, including members of the National Academy of Medicine and the National Academy of Sciences, and a distinguished two-time winner of the Albert E. Lasker Award in Medical Research. With an operating budget of more than $1.3 billion, the School of Medicine works closely in partnership with the University of Maryland Medical Center and Medical System to provide research-intensive, academic, and clinically based care for nearly 2 million patients each year. The School of Medicine has nearly $600 million in extramural funding, with most of its academic departments highly ranked among all medical schools in the nation in research funding. As one of the seven professional schools that make up the University of Maryland, Baltimore campus, the School of Medicine has a total population of nearly 9,000 faculty and staff, including 2,500 students, trainees, residents, and fellows. The combined School of Medicine and Medical System (“University of Maryland Medicine”) has an annual budget of over $6 billion and an economic impact of nearly $20 billion on the state and local community. The School of Medicine, which ranks as the 8th highest among public medical schools in research productivity (according to the Association of American Medical Colleges profile) is an innovator in translational medicine, with 606 active patents and 52 start-up companies. In the latest U.S. News & World Report ranking of the Best Medical Schools, published in 2021, the UM School of Medicine is ranked #9 among the 92 public medical schools in the U.S., and in the top 15 percent (#27) of all 192 public and private U.S. medical schools. The School of Medicine works locally, nationally, and globally, with research and treatment facilities in 36 countries around the world. Visit medschool.umaryland.edu