Libraries Open Historic Collections to AI Researchers, Boosting Machine Learning Capabilities

Harvard Leads the Charge in AI-Library Collaboration

In a groundbreaking move, Harvard University is releasing a vast collection of nearly one million books to AI researchers, marking a significant shift in how artificial intelligence systems are trained 1

1

. This initiative, part of the Harvard-based Institutional Data Initiative, is supported by tech giants Microsoft and OpenAI, and aims to provide AI developers with access to a rich trove of cultural, historical, and linguistic data 1

1

2

2

.

A Treasure Trove of Knowledge

The newly released dataset, dubbed Institutional Books 1.0, contains over 394 million scanned pages from books dating back to the 15th century, encompassing 254 languages 1

1

. This collection includes rare works such as a Korean painter's handwritten thoughts on horticulture from the 1400s, alongside a vast array of 19th-century literature on subjects ranging from philosophy to agriculture 3

3

.

Greg Leppert, executive director of the data initiative, emphasizes the importance of this collection: "A lot of the data that's been used in AI training has not come from original sources. This book collection goes all the way back to the physical copy that was scanned by the institutions that actually collected those items" 1

1

.

Addressing Copyright Concerns

The focus on public domain works is a strategic move to navigate the complex landscape of copyright issues that have plagued AI companies. Burton Davis, a deputy general counsel at Microsoft, notes, "It is a prudent decision to start with public domain data because that's less controversial right now than content that's still under copyright" 4

4

.

This approach comes as tech companies face legal challenges from authors and artists whose works have been used without consent to train AI models. Meta, for instance, is currently embroiled in a lawsuit with comedian Sarah Silverman and other authors over alleged copyright infringement 1

1

.

Expanding Beyond Harvard

The initiative extends beyond Harvard, with other institutions joining the effort. OpenAI has donated $50 million to a group of research institutions, including Oxford University's Bodleian Library, to digitize rare texts 1

1

. The Boston Public Library is also preparing to contribute old newspapers and government documents to the project 5

5

.

Jessica Chapel, chief of digital and online services at the Boston Public Library, emphasizes the mutual benefits of this collaboration: "OpenAI had this interest in massive amounts of training data. We have an interest in massive amounts of digital objects. So this is kind of just a case that things are aligning" 1

1

.

The Scale and Impact of the Data

Havard's new AI training collection boasts an estimated 242 billion tokens, a significant contribution to the field of machine learning. However, this pales in comparison to the most advanced AI systems. Meta's latest large language model, for example, was trained on more than 30 trillion tokens 1

1

.

A New Chapter in AI Development

This collaboration between libraries and tech companies represents a pivotal moment in AI development. By tapping into centuries of human knowledge preserved in library collections, AI researchers hope to create more accurate, reliable, and culturally informed systems. As Aristana Scourtas from Harvard Law School's Library Innovation Lab puts it, "We're trying to move some of the power from this current AI moment back to these institutions. Librarians have always been the stewards of data and the stewards of information" 1

1

.