Anubis: The Open-Source Bot Blocker Shielding Websites from AI Scrapers

The Rise of AI Web Scraping

In recent years, the proliferation of AI-powered web scrapers has become a significant concern for website owners and content creators. F5, an application delivery network company, reported that over half of all web visits now come from data scrapers, including those operated by major AI companies like OpenAI, Anthropic, Google, and Perplexity AI 1

1

. This trend has led to increased server loads, higher hosting costs, and concerns about intellectual property rights.

Introducing Anubis: The Open-Source Bot Blocker

In response to this challenge, Xe Iaso, a technical educator and part-time bot fighter, developed Anubis, an open-source program designed to stop AI bots in their tracks 1

1

. Anubis functions as a web AI firewall utility, requiring all incoming HTTP connections to pass through a series of challenges before reaching the actual website.

How Anubis Works

Anubis employs a unique approach to bot detection:

1. Proof-of-Work Challenges: Visitors must solve a computational puzzle, which is trivial for regular PCs but expensive for bots operating at scale 1

   1

   .
2. Browser Behavior Verification: The system checks whether visitors behave like real browsers, using JavaScript and cookies to verify authenticity 1

   1

   .
3. Transparent Operation: Unlike traditional CAPTCHAs, Anubis operates in the background, making it unnoticeable to most users [1](https://www.zdnet.com/article/this-open-source-bot-blocker-shields-your-site-from-pesky-ai-scrapers-heres-how()].

Key Features and Benefits

• Lightweight: Anubis uses less than 128 MB of RAM on the server side 1

  1

  .
• Customizable: Administrators can configure bot policy definitions to allow "good bots" like RSS feed readers 1

  1

  .
• Effective: Duke University reported blocking about 90% of unwanted traffic and over 4 million unwanted HTTP requests per day using Anubis 1

  1

  .

Adoption and Impact

Since its release in January 2025, Anubis has been downloaded over 200,000 times and adopted by various organizations, including GNOME, FFmpeg, and UNESCO 1

1

. It has helped numerous organizations avoid outages and reduce the burden of unwanted AI scraping.

Comparison to Other Solutions

While Anubis is not the only bot-blocking solution available, it stands out for its open-source nature and self-hostable design. Other similar tools include:

• Nepenthes: An LLM bot tarpit that generates endless pages of nonsense text 2

  2

  .
• Quixotic and Linkmaze: Tools that work similarly to Nepenthes 2

  2

  .
• TollBit: A commercial bot-blocking solution 2

  2

  .

Future Developments and Challenges

Iaso is currently working on a non-JavaScript version of Anubis to address concerns about running JavaScript for security and privacy reasons 1

1

. However, this development is expected to take some time due to its complexity.

As the battle between website owners and AI scrapers continues, tools like Anubis represent an important step in protecting online content and infrastructure. However, the ongoing challenge will be to balance effective bot blocking with maintaining accessibility for legitimate users and beneficial web services.