Cloudflare introduces new tools for website owners to protect themselves from web crawlers and get a fair price for their content.

The internet is teeming with web crawlers looking for interesting content to train AI models. The collected content is used without prior consent, let alone paying a fair price for it. Cloudflare gives website owners the ability to protect their content from web crawlers.

After introducing a tool to disable web crawlers in July, Cloudflare is now taking a more intermediate approach. The internet company is announcing AI Audit, a collection of free tools to help customers understand how AI models are using their content. Cloudflare’s tool can see where each crawler is coming from, who owns it, and how often you get visits from each web crawler, among other things.

With this insight, companies can decide whether or not to grant access to AI developers and set a “fair price” for using their content for model training and other purposes.

Restore balance

The use of web crawlers has become a controversial topic in the wake of the rise of generative AI. ChatGPT knows so much because OpenAI scoured the internet for information, Dall-E learned its drawing skills from existing photos and artwork. To this end, it “scraped” publicly available content from the internet without regard to copyright.

With AI Audit, Cloudflare says it wants to restore the balance between (small) content creators and (large) technology companies. “If you don’t compensate creators in some way, then they’ll stop creating, and that’s the part that needs to be fixed. If we don’t do that, the quality and availability of online content will only decline,” says CEO Matthew Prince in an interview with TechCrunch.

Some websites are taking matters into their own hands. Reddit is banning all search engines from companies that do not pay for the content on the social media platform. It is selling its skin very expensively: Google has signed a contract with Reddit for sixty million dollars a year.