ChatGPT Like AI Model Shows Promise in Cancer Treatment Decision-Making

ChatGPT-like model can diagnose cancer, guide treatment choice, predict survival across multiple cancer types

Scientists at Harvard Medical School have designed a versatile, ChatGPT-like AI model capable of performing an array of diagnostic tasks across multiple forms of cancers. The new AI system, described Sept. 4 in Nature, goes a step beyond many current AI approaches to cancer diagnosis, the researchers said. Current AI systems are typically trained to perform specific tasks -- such as detecting cancer presence or predicting a tumor's genetic profile -- and they tend to work only in a handful of cancer types. By contrast, the new model can perform a wide array of tasks and was tested on 19 cancer types, giving it a flexibility like that of large language models such as ChatGPT. While other foundation AI models for medical diagnosis based on pathology images have emerged recently, this is believed to be the first to predict patient outcomes and validate them across several international patient groups. "Our ambition was to create a nimble, versatile ChatGPT-like AI platform that can perform a broad range of cancer evaluation tasks," said study senior author Kun-Hsing Yu, assistant professor of biomedical informatics in the Blavatnik Institute at Harvard Medical School. "Our model turned out to be very useful across multiple tasks related to cancer detection, prognosis, and treatment response across multiple cancers." The AI model, which works by reading digital slides of tumor tissues, detects cancer cells and predicts a tumor's molecular profile based on cellular features seen on the image with superior accuracy to most current AI systems. It can forecast patient survival across multiple cancer types and accurately pinpoint features in the tissue that surrounds a tumor -- also known as the tumor microenvironment -- that are related to a patient's response to standard treatments, including surgery, chemotherapy, radiation, and immunotherapy. Finally, the team said, the tool appears capable of generating novel insights -- it identified specific tumor characteristics previously not known to be linked to patient survival. The findings, the research team said, add to growing evidence that AI-powered approaches can enhance clinicians' ability to evaluate cancers efficiently and accurately, including the identification of patients who might not respond well to standard cancer therapies. "If validated further and deployed widely, our approach, and approaches similar to ours, could identify early on cancer patients who may benefit from experimental treatments targeting certain molecular variations, a capability that is not uniformly available across the world," Yu said. The team's latest work builds on Yu's previous research in AI systems for the evaluation of colon cancer and brain tumors. These earlier studies demonstrated the feasibility of the approach within specific cancer types and specific tasks. The new model, called CHIEF (Clinical Histopathology Imaging Evaluation Foundation), was trained on 15 million unlabeled images chunked into sections of interest. The tool was then trained further on 60,000 whole-slide images of tissues including lung, breast, prostate, colorectal, stomach, esophageal, kidney, brain, liver, thyroid, pancreatic, cervical, uterine, ovarian, testicular, skin, soft tissue, adrenal gland, and bladder. Training the model to look both at specific sections of an image and the whole image allowed it to relate specific changes in one region to the overall context. This approach, the researchers said, enabled CHIEF to interpret an image more holistically by considering a broader context, instead of just focusing on a particular region. Following training, the team tested CHIEF's performance on more than 19,400 whole-slide images from 32 independent datasets collected from 24 hospitals and patient cohorts across the globe. Overall, CHIEF outperformed other state-of-the-art AI methods by up to 36% on the following tasks: cancer cell detection, tumor origin identification, predicting patient outcomes, and identifying the presence of genes and DNA patterns related to treatment response. Because of its versatile training, CHIEF performed equally well no matter how the tumor cells were obtained -- whether via biopsy or through surgical excision. And it was just as accurate, regardless of the technique used to digitize the cancer cell samples. This adaptability, the researchers said, renders CHIEF usable across different clinical settings and represents an important step beyond current models that tend to perform well only when reading tissues obtained through specific techniques. CHIEF achieved nearly 94% accuracy in cancer detection and significantly outperformed current AI approaches across 15 datasets containing 11 cancer types. In five biopsy datasets collected from independent cohorts, CHIEF achieved 96% accuracy across multiple cancer types including esophagus, stomach, colon, and prostate. When the researchers tested CHIEF on previously unseen slides from surgically removed tumors of the colon, lung, breast, endometrium, and cervix, the model performed with more than 90% accuracy. A tumor's genetic makeup holds critical clues to determine its future behavior and optimal treatments. To get this information, oncologists order DNA sequencing of tumor samples, but such detailed genomic profiling of cancer tissues is not done routinely nor uniformly across the world due to the cost and time involved in sending samples to specialized DNA sequencing labs. Even in well-resourced regions, the process could take several weeks. It's a gap that AI could fill, Yu said. Quickly identifying cellular patterns on an image suggestive of specific genomic aberrations could offer a quick and cost-effective alternative to genomic sequencing, the researchers said. CHIEF outperformed current AI methods for predicting genomic variations in a tumor by looking at the microscopic slides. This new AI approach successfully identified features associated with several important genes related to cancer growth and suppression, and it predicted key genetic mutations related to how well a tumor might respond to various standard therapies. CHIEF also detected specific DNA patterns related to how well a colon tumor might respond to a form of immunotherapy called immune checkpoint blockade. When looking at whole-tissue images, CHIEF identified mutations in 54 commonly mutated cancer genes with an overall accuracy of more than 70%, outperforming the current state-of-the-art AI method for genomic cancer prediction. Its accuracy was greater for specific genes in specific cancer types. The team also tested CHIEF on its ability to predict mutations linked with response to FDA-approved targeted therapies across 18 genes spanning 15 anatomic sites. CHIEF attained high accuracy in multiple cancer types, including 96% in detecting a mutation in a gene called EZH2 common in a blood cancer called diffuse large B-cell lymphoma. It achieved 89% for BRAF gene mutation in thyroid cancer, and 91% for NTRK1 gene mutation in head and neck cancers. CHIEF successfully predicted patient survival based on tumor histopathology images obtained at the time of initial diagnosis. In all cancer types and all patient groups under study, CHIEF distinguished patients with longer-term survival from those with shorter-term survival. CHIEF outperformed other models by 8%. And in patients with more advanced cancers, CHIEF outperformed other AI models by 10%. In all, CHIEF's ability to predict high versus low death risk was tested and confirmed across patient samples from 17 different institutions. The model identified tell-tale patterns on images related to tumor aggressiveness and patient survival. To visualize these areas of interest, CHIEF generated heat maps on an image. When human pathologists analyzed these AI-derived hot spots, they saw intriguing signals reflecting interactions between cancer cells and surrounding tissues. One such feature was the presence of greater numbers of immune cells in areas of the tumor in longer-term survivors, compared with shorter-term survivors. That finding, Yu noted, makes sense because a greater presence of immune cells may indicate the immune system has been activated to attack the tumor. When looking at the tumors of shorter-term survivors, CHIEF identified regions of interest marked by the abnormal size ratios between various cell components, more atypical features on the nuclei of cells, weak connections between cells, and less presence of connective tissue in the area surrounding the tumor. These tumors also had a greater presence of dying cells around them. For example, in breast tumors, CHIEF pinpointed as an area of interest the presence of necrosis -- or cell death -- inside the tissues. On the flip side, breast cancers with higher survival rates were more likely to have preserved cellular architecture resembling healthy tissues. The visual features and zones of interest related to survival varied by cancer type, the team noted. The researchers said they plan to refine CHIEF's performance and augment its capabilities by:

A New Artificial Intelligence Tool for Cancer | Newswise

Newswise -- Scientists at Harvard Medical School have designed a versatile, ChatGPT-like AI model capable of performing an array of diagnostic tasks across multiple forms of cancers. The new AI system, described Sept. 4 in Nature, goes a step beyond many current AI approaches to cancer diagnosis, the researchers said. (DOI 10.1038/s41586-024-07894-z) Current AI systems are typically trained to perform specific tasks -- such as detecting cancer presence or predicting a tumor's genetic profile -- and they tend to work only in a handful of cancer types. By contrast, the new model can perform a wide array of tasks and was tested on 19 cancer types, giving it a flexibility like that of large language models such as ChatGPT. While other foundation AI models for medical diagnosis based on pathology images have emerged recently, this is believed to be the first to predict patient outcomes and validate them across several international patient groups. "Our ambition was to create a nimble, versatile ChatGPT-like AI platform that can perform a broad range of cancer evaluation tasks," said study senior author Kun-Hsing Yu, assistant professor of biomedical informatics in the Blavatnik Institute at Harvard Medical School. "Our model turned out to be very useful across multiple tasks related to cancer detection, prognosis, and treatment response across multiple cancers." The AI model, which works by reading digital slides of tumor tissues, detects cancer cells and predicts a tumor's molecular profile based on cellular features seen on the image with superior accuracy to most current AI systems. It can forecast patient survival across multiple cancer types and accurately pinpoint features in the tissue that surrounds a tumor -- also known as the tumor microenvironment -- that are related to a patient's response to standard treatments, including surgery, chemotherapy, radiation, and immunotherapy. Finally, the team said, the tool appears capable of generating novel insights -- it identified specific tumor characteristics previously not known to be linked to patient survival. The findings, the research team said, add to growing evidence that AI-powered approaches can enhance clinicians' ability to evaluate cancers efficiently and accurately, including the identification of patients who might not respond well to standard cancer therapies. "If validated further and deployed widely, our approach, and approaches similar to ours, could identify early on cancer patients who may benefit from experimental treatments targeting certain molecular variations, a capability that is not uniformly available across the world," Yu said. Training and performance The team's latest work builds on Yu's previous research in AI systems for the evaluation of colon cancer and brain tumors. These earlier studies demonstrated the feasibility of the approach within specific cancer types and specific tasks. The new model, called CHIEF (Clinical Histopathology Imaging Evaluation Foundation), was trained on 15 million unlabeled images chunked into sections of interest. The tool was then trained further on 60,000 whole-slide images of tissues including lung, breast, prostate, colorectal, stomach, esophageal, kidney, brain, liver, thyroid, pancreatic, cervical, uterine, ovarian, testicular, skin, soft tissue, adrenal gland, and bladder. Training the model to look both at specific sections of an image and the whole image allowed it to relate specific changes in one region to the overall context. This approach, the researchers said, enabled CHIEF to interpret an image more holistically by considering a broader context, instead of just focusing on a particular region. Following training, the team tested CHIEF's performance on more than 19,400 whole-slide images from 32 independent datasets collected from 24 hospitals and patient cohorts across the globe. Overall, CHIEF outperformed other state-of-the-art AI methods by up to 36 percent on the following tasks: cancer cell detection, tumor origin identification, predicting patient outcomes, and identifying the presence of genes and DNA patterns related to treatment response. Because of its versatile training, CHIEF performed equally well no matter how the tumor cells were obtained -- whether via biopsy or through surgical excision. And it was just as accurate, regardless of the technique used to digitize the cancer cell samples. This adaptability, the researchers said, renders CHIEF usable across different clinical settings and represents an important step beyond current models that tend to perform well only when reading tissues obtained through specific techniques. Cancer detection CHIEF achieved nearly 94 percent accuracy in cancer detection and significantly outperformed current AI approaches across 15 datasets containing 11 cancer types. In five biopsy datasets collected from independent cohorts, CHIEF achieved 96 percent accuracy across multiple cancer types including esophagus, stomach, colon, and prostate. When the researchers tested CHIEF on previously unseen slides from surgically removed tumors of the colon, lung, breast, endometrium, and cervix, the model performed with more than 90 percent accuracy. Predicting tumors' molecular profiles A tumor's genetic makeup holds critical clues to determine its future behavior and optimal treatments. To get this information, oncologists order DNA sequencing of tumor samples, but such detailed genomic profiling of cancer tissues is not done routinely nor uniformly across the world due to the cost and time involved in sending samples to specialized DNA sequencing labs. Even in well-resourced regions, the process could take several weeks. It's a gap that AI could fill, Yu said. Quickly identifying cellular patterns on an image suggestive of specific genomic aberrations could offer a quick and cost-effective alternative to genomic sequencing, the researchers said. CHIEF outperformed current AI methods for predicting genomic variations in a tumor by looking at the microscopic slides. This new AI approach successfully identified features associated with several important genes related to cancer growth and suppression, and it predicted key genetic mutations related to how well a tumor might respond to various standard therapies. CHIEF also detected specific DNA patterns related to how well a colon tumor might respond to a form of immunotherapy called immune checkpoint blockade. When looking at whole-tissue images, CHIEF identified mutations in 54 commonly mutated cancer genes with an overall accuracy of more than 70 percent, outperforming the current state-of-the-art AI method for genomic cancer prediction. Its accuracy was greater for specific genes in specific cancer types. The team also tested CHIEF on its ability to predict mutations linked with response to FDA-approved targeted therapies across 18 genes spanning 15 anatomic sites. CHIEF attained high accuracy in multiple cancer types, including 96 percent in detecting a mutation in a gene called EZH2 common in a blood cancer called diffuse large B-cell lymphoma. It achieved 89 percent for BRAF gene mutation in thyroid cancer, and 91 percent for NTRK1 gene mutation in head and neck cancers. Predicting patient survival CHIEF successfully predicted patient survival based on tumor histopathology images obtained at the time of initial diagnosis. In all cancer types and all patient groups under study, CHIEF distinguished patients with longer-term survival from those with shorter-term survival. CHIEF outperformed other models by 8 percent. And in patients with more advanced cancers, CHIEF outperformed other AI models by 10 percent. In all, CHIEF's ability to predict high versus low death risk was tested and confirmed across patient samples from 17 different institutions. Extracting novel insights about tumor behavior The model identified tell-tale patterns on images related to tumor aggressiveness and patient survival. To visualize these areas of interest, CHIEF generated heat maps on an image. When human pathologists analyzed these AI-derived hot spots, they saw intriguing signals reflecting interactions between cancer cells and surrounding tissues. One such feature was the presence of greater numbers of immune cells in areas of the tumor in longer-term survivors, compared with shorter-term survivors. That finding, Yu noted, makes sense because a greater presence of immune cells may indicate the immune system has been activated to attack the tumor. When looking at the tumors of shorter-term survivors, CHIEF identified regions of interest marked by the abnormal size ratios between various cell components, more atypical features on the nuclei of cells, weak connections between cells, and less presence of connective tissue in the area surrounding the tumor. These tumors also had a greater presence of dying cells around them. For example, in breast tumors, CHIEF pinpointed as an area of interest the presence of necrosis -- or cell death -- inside the tissues. On the flip side, breast cancers with higher survival rates were more likely to have preserved cellular architecture resembling heathy tissues. The visual features and zones of interest related to survival varied by cancer type, the team noted. Next steps The researchers said they plan to refine CHIEF's performance and augment its capabilities by: Authorship, funding, disclosures Co-authors included Xiyue Wang, Junhan Zhao, Eliana Marostica, Wei Yuan, Jietian Jin, Jiayu Zhang, Ruijiang Li, Hongping Tang, Kanran Wang, Yu Li, Fang Wang, Yulong Peng, Junyou Zhu, Jing Zhang, Christopher R. Jackson, Jun Zhang, Deborah Dillon, Nancy U. Lin, Lynette Sholl, Thomas Denize, David Meredith, Keith L. Ligon, Sabina Signoretti, Shuji Ogino, Jeffrey A. Golden, MacLean P. Nasrallah, Xiao Han, Sen Yang. The work was in part supported by the National Institute of General Medical Sciences grant R35GM142879, the Department of Defense Peer Reviewed Cancer Research Program Career Development Award HT9425-23-1-0523, the Google Research Scholar Award, the Harvard Medical School Dean's Innovation Award, and the Blavatnik Center for Computational Biomedicine Award.

AI breakthrough raises hopes for better cancer diagnosis

A new artificial intelligence foundation model can accurately detect multiple cancer types, assess treatments and predict survival rates, in the latest advance in medical diagnosis driven by the technology. The model -- known as "Chief" -- is a breakthrough because of the breadth of tumours it can analyse and its capacity to predict outcomes for patients, its Harvard Medical School inventors say. Chief highlights how AI has powered improvements in diagnostic techniques based on images, in part because it is able to spot the significance of features even an experienced human eye may miss. "Our ambition was to create a nimble, versatile ChatGPT-like AI platform that can perform a broad range of cancer evaluation tasks," said Kun-Hsing Yu, assistant professor of biomedical informatics at Harvard Medical School's Blavatnik Institute. "Our model turned out to be very useful across multiple tasks related to cancer detection, prognosis, and treatment response across multiple cancers." While recent breakthroughs in AI have led to fears about the technology's abuse, optimists have argued it also has the power to produce long-term benefits to humanity in fields such as medicine and climate science. Chief, described in a paper published in Nature today, works by reading digital slides of tumour tissues. It was trained on 15mn unlabelled sections of images and then 60,000 whole slide images of tissues, covering 19 different cancers. The idea was to make sure Chief could relate detailed changes in one region of tissue to its wider context, the researchers said. They tested its performance on almost 20,000 whole-slide images from 24 hospitals and patient cohorts around the world. Chief outperformed other AI diagnostic methods by up to 36 per cent in detecting cancer cells, predicting patient outcomes, and identifying tumour origins and the presence of genetic patterns related to treatment response, the paper said. Unlike some other current models, it had the versatility to maintain its performance regardless of the techniques used to obtain and digitise the tumour cells, they added. Chief showed overall accuracy of almost 94 per cent for cancer detection, rising to 96 per cent for oesophagus, stomach, colon and prostate tumours. Its ability to link tumour cell patterns to specific genomic aberrations could help suggest best treatments without the need for costly and slow DNA sequencing, the scientists suggested. The model offered further revealing information about tissues surrounding tumours, including the presence of a greater number of immune cells in long-term cancer survivors versus those who died sooner, the paper said. If Chief and similar approaches are validated by further research, they could be used to "identify early on cancer patients who may benefit from experimental treatments targeting certain molecular variations", including in countries where that isn't currently done, Yu said. AI models are proving an increasingly useful ally to medical professionals in the field of imaging, because of their speed and pattern-spotting ability. While still imperfect, they can be helpful in triage, as a second opinion or to generate insights a doctor may have overlooked or not know about. Chief appeared to be an important new pan-cancer tool in a growing field of diagnostic foundational AI models, said Professor Eric Topol, founder and director of the Scripps Research Translational Institute in California. In April, Harvard Medical School researchers at Boston's Brigham and Women's Hospital announced two models -- known as Uni and Conch -- to read, interpret and classify microscopy slides from patient tissues. They showed good results on diagnostic tasks ranging from disease detection to organ transplant assessment, as well as showing some ability to identify new and rare conditions. The evolving new AI foundation diagnostic models promised to "provide exceptional insight from whole slide images, including improving accuracy of diagnosis and prognosis", Topol said.