Study Evaluates ChatGPT's Accuracy in Answering Blood Cancer Questions

ChatGPT's Performance in Hematology-Oncology

A new study published in the journal Future Science OA has evaluated the accuracy of ChatGPT 3.5 in answering questions related to blood cancer 1

1

. The research, conducted by a team led by Dr. Justin Taylor from Sylvester Comprehensive Cancer Center at the University of Miami Miller School of Medicine, aimed to assess the chatbot's performance in a rapidly evolving medical field.

Study Methodology and Findings

The study, carried out in July 2024, posed 10 questions to ChatGPT 3.5, mimicking typical patient inquiries during cancer treatment 2

2

. Four anonymous hematology-oncology physicians evaluated the responses on a scale of 1 to 5. The results showed:

• ChatGPT performed better on general cancer questions (average score 3.38)
• It struggled with questions about newer therapies (average score 3.06)
• No answer received the highest score of 5 from any evaluator

Implications for Patient Use

Dr. Taylor cautioned patients about relying solely on AI-generated medical information, especially for specific cancer types and treatments 1

1

. The study concluded that "physician oversight remains essential for vetting AI-generated medical information before patient use."

Limitations and Future Perspectives

The research acknowledged limitations, including:

• Only testing ChatGPT 3.5, which had a 2021 data cutoff
• Not evaluating other LLMs or newer versions of ChatGPT

Despite these limitations, Dr. Taylor emphasized that the message of caution still applies to newer AI models 2

2

. He noted that while chatbots can't provide nuanced, personalized understanding like oncologists, they may still be useful for:

• Helping patients prepare for medical visits
• Formulating questions for physicians
• Directing patients to primary sources with accurate information

Broader AI Initiatives in Medicine

The study aligns with various AI initiatives at Sylvester and the Miller School of Medicine, including:

• AI tools for summarizing patient encounters and form-filling
• An AI-powered system for brain tumor diagnosis during optical imaging
• A machine learning risk-prediction model for multiple myeloma patients

The Miller School has also launched educational programs on AI in medicine, including an elective course for medical students and an AI ethics course for Spanish-speaking medical professionals in Latin America 1

1

.

Future Research

Dr. Taylor and his colleagues plan to reassess the accuracy of newer ChatGPT versions within the next year or two, acknowledging the rapid evolution of LLMs and associated technologies 2

2

.

The study, titled "ChatGPT's Role in the Rapidly Evolving Hematologic Cancer Landscape," involved collaborators from Florida State University College of Medicine, the University of Vermont, and Florida Cancer Specialists 1

1

2

2

.