2 Sources
[1]
AI helps researchers investigate how thousands of medical studies share code
An MIT and Dana-Farber Institute-led team used large language models to examine nearly 4,000 studies and hundreds of code repositories, revealing gaps in the information needed to scrutinize and reuse published research. The data and code that underpin a research study can tell us what was
[2]
Public code is not always reusable: Review uncovers gaps in medical study documentation
The data and code that underpin a research study can tell us what was actually done, and how. In a medical study, for example, how were patients selected for inclusion? How were missing values handled? How were the results evaluated? Access to a study's data and code helps researchers to answer
Share
Copy Link
Only about 1 in 8 clinical prediction studies reported sharing the code behind their models, and even the code that did surface was often not usable by other researchers. That is the result of a review of nearly 4,000 studies, led by MIT and Dana-Farber Cancer Institute researchers and published open-access in Nature Medicine, which used large language models to assess hundreds of linked code repositories
1
2
.The stakes come from what these models do. Clinical prediction models help inform decisions about patient care, so researchers need a way to check how they were built and tested
1
. A paper alone may not be enough for that. Tom Pollard, an MIT research scientist and the study's senior author, explains that research involves many choices about data preparation, modeling and evaluation. "The choices we make affect the results, and a methods section in a paper can only capture so much detail," he said1
. Code is how an outsider sees those choices.The team used a large language model to screen papers, pick out code-availability statements and extract links. Software then retrieved the repositories and checked them against documentation completeness, software requirements and terms of reuse
1
. "The real benefit of LLMs is their ability to process unstructured information such as text," Pollard said1
.
Source: MIT
Two human researchers independently checked 500 papers plus a separate sample of repositories. Agreement with the model was stronger for concrete features, like whether a license file exists, than for judgments about whether documentation is adequate. The authors themselves flag that classification errors remain possible, and they point to the overall patterns rather than the exact counts
1
2
.The review found two gaps. Sharing is rare. And code that is made public does not reliably come with the documentation someone else needs to run it
1
. First author Thomas Sounack said some repositories were "extremely well-organized and -documented, while others gave almost no guidance at all"1
. The useful standard, in his words, is "giving enough information to understand what the code does and be able to run it to reproduce the results reported in the paper"1
.The sample has limits worth knowing: the studies reviewed were those citing earlier reporting checklists for prediction models, and their full texts were openly available on PubMed Central. The findings describe that open-access slice of the literature, not all medical research
1
.Across journals, the share of studies sharing code ran from none at all to above 70 percent
1
2
. Some journals with low rates already had written code-sharing policies, so policies alone may not be enough to change behavior1
2
. There is currently little guidance on what a well-organized, well-described code collection even looks like1
.The findings are now informing TRIPOD-Code, a reporting guideline still under development that extends those earlier checklists
1
. The field spans medicine, statistics and computer science, and those disciplines may expect different things from a code collection, so the guideline is aiming at consensus1
. "We want to give researchers practical guidance on how to share code, and give journals clearer expectations to communicate to authors," Pollard said1
.Related Stories
Summarized by
Navi
23 Oct 2025•Science and Research

11 Nov 2026•Science and Research

29 Oct 2024•Science and Research
