Millions of websites will now be able to block AI bots from accessing their content without permission thanks to a new system being rolled out by internet infrastructure firm, Cloudflare. Eventually, sites will be able to ask for payment from AI firms in return for having their content scraped.
Cloudflare’s tech targets AI firm bots – also known as crawlers – which are programs that explore the web, indexing and collecting data as they go. They are important to the way AI firms build, train and operate their systems. So far, Cloudflare says its tech is active on a million websites.
Roger Lynch, chief executive of Condé Nast, whose print titles include GQ, Vogue, and The New Yorker, said the move was “a game-changer” for publishers. “This is a critical step toward creating a fair value exchange on the Internet that protects creators, supports quality journalism and holds AI companies accountable”, he wrote in a statement. However, other experts say stronger legal protections will still be needed.
Initially the system will apply by default to new users of Cloudflare services, plus sites that participated in an earlier effort to block crawlers. Many publishers accuse AI firms of using their content without permission.
Cloudflare argues AI breaks the unwritten agreement between publishers and crawlers. AI crawlers, it argues, collect content like text, articles, and images to generate answers, without sending visitors to the original source—depriving content creators of revenue. “If the Internet is going to survive the age of AI, we need to give publishers the control they deserve and build a new economic model that works for everyone,” wrote the firm’s chief executive Matthew Prince.
To that end the company is developing a “Pay Per Crawl” system, which would give content creators the option to request payment from AI companies for utilising their original content.
According to Cloudflare there has been an explosion of AI bot activity. “AI Crawlers generate more than 50 billion requests to the Cloudflare network every day”, the company wrote in March. And there is growing concern that some AI crawlers are disregarding existing protocols for excluding bots. In an effort to counter the worst offenders Cloudflare previously developed a system where the worst miscreants would be sent to a “Labyrinth” of web pages filled with AI generated junk.
The new system attempts to use technology to protect the content of websites and to give sites the option to charge AI firms a fee to access it.
Source: BBC News
Image Credit: Cloudflare
