News Publishers Block AI Training Access to Web Archives Amid Copyright Infringement Concerns

News Publishers Take Action Against Web Archive Used for AI

Major news publishers are mounting a coordinated effort to block web archives that AI companies exploit to train large language models without permission or compensation. Over 240 news organizations across nine countries, including The New York Times, CNN, USA Today, and The Guardian, are now restricting access to their content stored in digital archives 2

2

3

3

. The News/Media Alliance, representing 20 publishers, sent a letter to Common Crawl demanding the nonprofit honor opt-out requests and prohibit unauthorized use of their work for AI training purposes .

Danielle Coffey, president and CEO of News/Media Alliance, described the situation bluntly: "We do view this type of use of our content without permission as a copyright violation" . The alliance discovered the widespread use of Common Crawl data in AI training approximately three years ago, raising immediate concerns about copyright infringement.

How AI Companies Use Copyrighted Content From Archives

Common Crawl, founded in 2007, operates a bot that traverses the web to create a digital archive accessible to anyone . AI companies including OpenAI, Google, and Meta Platforms have tapped this trove of archived news content to develop chatbots like ChatGPT. The nonprofit has received donations from OpenAI and Anthropic, raising questions about potential conflicts of interest .

Archived news content offers AI developers exactly what they need: structured, dated, attributed, high-quality writing accumulated over decades 2

2

. The Internet Archive's Wayback Machine, which has preserved more than one trillion web pages since 1996, makes enormous quantities of content accessible via API and URL interface—an ideal source for model training pipelines 2

2

. A 2023 Washington Post analysis confirmed that data from the Internet Archive appeared in major AI training datasets 2

2

.

Legal Battles Over AI and Publisher Concerns

News outlets worry that chatbots and AI-powered search results will steal their content, reduce web traffic, and cost them valuable advertising revenue. The New York Times sued Microsoft and OpenAI over alleged copyright infringement causing billions of dollars in damages, claiming news articles were scraped and copied almost verbatim in chatbot responses . Graham James, a Times spokesperson, stated: "The issue is that Times content on the Internet Archive is being used by AI companies in violation of copyright law to directly compete with us" 2

2

.

For publishers engaged in active lawsuits against OpenAI, Perplexity, and others, web archives represent a significant gap in their defenses 2

2

. Meanwhile, some companies including Axel Springer, Condé Nast, and Hearst have signed lucrative licensing deals that allow controlled use of their content in AI tools .

Blocking Internet Archives Creates Collateral Damage

At least 23 major news publications are blocking ia_archiverbot, the main web crawler the Internet Archive uses for the Wayback Machine, according to analysis by AI-detection startup Originality AI 2

2

. USA Today Co., the largest newspaper publisher in the US, accounts for a substantial share of blocked sites, effectively removing hundreds of local publications from historical records 2

2

3

3

.

Mark Graham, the Wayback Machine's director, characterized the situation plainly: "We are collateral damage" 2

2

3

3

. The Archive has implemented its own protective measures, including rate-limiting bulk downloads and blocking large-scale automated extraction from certain sites 2

2

3

3

. Graham argues that publishers' rationale for blocking crawlers is "unfounded," since the risk comes from AI companies accessing archived material through interfaces the Archive controls, not from preservation activities themselves 2

2

.

Threats to Public Record and Accountability

The consequences of blocking web crawlers extend far beyond preventing AI training. When news articles are no longer archived, they become editable without accountability. Publishers can quietly amend stories after publication—correcting errors, softening claims, or removing quotes—and the Wayback Machine has been the primary tool journalists use to document those changes 2

2

3

3

. Courts cite the Archive, historians treat it as a primary source, and journalists rely on it for preservation of the public record 2

2

.

The Electronic Frontier Foundation's Joe Mullin framed the stakes clearly: "The Internet Archive often becomes the only source for seeing those changes. There are real disputes over AI training that must be resolved in courts. But sacrificing the public record to fight those battles would be a profound, and possibly irreversible, mistake" 2

2

. Fight for the Future has launched a petition, already signed by 100 current journalists, protesting the blocking at a time when public records and history are increasingly contested 3

3

.

Publishers Seek Middle Ground Solutions

Some organizations are taking more measured approaches. The Guardian limited rather than fully blocked Archive access after discovering it was a frequent crawler 2

2

. Robert Hahn, head of business affairs at The Guardian, expressed particular concern about APIs: "A lot of these AI businesses are looking for readily available, structured databases of content. The Internet Archive's API would have been an obvious place to plug their own machines into and suck out the IP" 2

2

.

The Archive has been actively working with publishers to find acceptable compromises involving limited access rather than hard blocks 2

2

3

3

. Common Crawl established an opt-out registry allowing publishers to request content exclusion from web crawls, though a November investigation by The Atlantic claimed Common Crawl doesn't always honor these opt-out requests and circumvents paywalls—allegations the nonprofit denied . As legal battles continue and AI capabilities expand, publishers face difficult choices between protecting their intellectual property and maintaining historical records that serve public interest.