More than three dozen businesses involved in intellectual property, including Universal Music Group, have applauded a move by internet infrastructure company Cloudlfare that blocks AI companies from sucking up the world’s digital content – and in the process creates a new AI licensing market.
As of Tuesday (July 1), AI web crawlers are blocked by default from accessing the content on websites that use Cloudflare infrastructure. As the company provides web security services to nearly 20% of all sites on the internet, that could significantly frustrate the efforts of AI companies seeking to download copyrighted content to train their AI models.
And that’s not all: Website owners will also have the ability to choose which crawlers are allowed to access their content, and adopt a “pay-per-crawl” business model that sets out how much a web crawler has to pay to access their content.
This means content owners will be able to use an automated system to earn an income from any AI company willing to pay the rate it sets, or to negotiate deals with specific AI companies for access to their content.
Web crawlers are programs that scan the vast quantities of websites around the world one by one. They were originally used to create the databases that allow search engines to work, but in recent years they have been repurposed to help AI companies collect the vast amounts of text, images and other content they use to train AI – often without the permission of content owners.
With the public now turning to AI for information, problem-solving and even entertainment, the result has been a massive drop-off in traffic to the online sites owned by content creators, in what has been dubbed the “zero-click” phenomenon.
“If the Internet is going to survive the age of AI, we need to give publishers the control they deserve and build a new economic model that works for everyone.”
Matthew Prince, Cloudflare
“If the Internet is going to survive the age of AI, we need to give publishers the control they deserve and build a new economic model that works for everyone – creators, consumers, tomorrow’s AI founders, and the future of the web itself,” said Matthew Prince, co-founder and CEO of Cloudflare.
“Original content is what makes the internet one of the greatest inventions in the last century, and it’s essential that creators continue making it. AI crawlers have been scraping content without limits. Our goal is to put the power back in the hands of creators, while still helping AI companies innovate. This is about safeguarding the future of a free and vibrant Internet with a new model that works for everyone.”
Cloudflare’s move has the backing of numerous businesses and organizations involved in IP-related industries, including the News/Media Alliance and a number of news and magazine publishers, including the Associated Press, Conde Nast, Fortune, Sky News Group, and Time, among others.
It also has the backing of the world’s largest music company, Universal Music Group (UMG).
“We welcome this new initiative from Cloudflare, that will help address the indiscriminate, disruptive, and unauthorized scraping of both creative and commercial IP by AI model developers and support new licensing,” UMG Chief Operating Officer Boyd Muir said.
“At UMG, we have always embraced innovation and new technologies, and firmly believe that AI, when used ethically, transparently, and respectfully of copyright and human creativity, has the opportunity to introduce significant new avenues for creativity and future monetization.”
“We welcome this new initiative from Cloudflare, that will help address the indiscriminate, disruptive, and unauthorized scraping of both creative and commercial IP by AI model developers and support new licensing.”
Boyd Muir, Universal Music Group
As an internet security company, Cloudflare is well positioned to fight unwanted AI web crawlers. The company already has a bot verification system that allows web crawlers to identify themselves, their purpose, and the company they work for.
And for those web crawlers that won’t cooperate, the company can deploy the same sorts of tools it uses to fight distributed denial-of-service (DDoS) attacks – attempts by malicious actors to cripple websites by overloading them with a flood of internet traffic.
“A web crawler that is going across the internet looking for the latest content is just another type of bot – so all of our work to understand traffic and network patterns for the clearly malicious bots helps us understand what a crawler is doing,” Will Allen, Cloudflare’s head of AI privacy, control, and media products, told MIT Technology Review.
Nonetheless, some individuals in the field of AI caution that Cloudflare’s approach could harm legitimate research being carried out in the online world.
“Not all AI systems compete with all web publishers. Not all AI systems are commercial,” Shayne Longpre, a PhD candidate at the MIT Media Lab, told MIT Technology Review. “Personal use and open research shouldn’t be sacrificed here.”Music Business Worldwide