Each time you ask for an answer from AI, whether through a Google search bar or ChatGPT prompt, the response is generated from content lifted by AI scrapers across the Internet or even in print.
This sounds fine until you realise companies such as Google or OpenAI paid nothing to the original creators of the content, whether it’s a tip to find the best eateries in Tokyo or a summary of the latest data centre trends across Asia-Pacific.
The information is public and the AI is merely looking through all these sources individually and summing things up for you, so the argument goes from AI companies.
Unfortunately, this is unsustainable because AI scrapers undermine today’s content business model. Publishers from the New York Times to small independent websites like Techgoondu depend on people to consume their actual content – not derivatives spit out by AI – to keep things going.
As fewer people visit the actual websites and video channels of content creators and publishers, they lose revenue that could be gleaned from online advertising and subscriptions.
And that is already happening, with online traffic dropping drastically in some cases. AI-generated answers are now replacing eyeballs on these content websites. Eventually, without good content for AI to learn from, it becomes less accurate and useful.
One way forward is what Cloudflare proposed this week. The Internet company, which handles about 20 per cent of the online traffic in the world, says it is blocking AI bots from its customers’ websites by default.
It is working on a payment model where these customers can be compensated for creating content that AI companies find useful and thus should be paying to train their models. Want to scrape and exploit the content? Pay up.
The move is a bold one that aims to make things fairer and more sustainable. As Cloudflare chief executive Matthew Prince says, “the incentives for content creation are dead” and his initiative is a shot at fairer trade.
More than a defensive move, the change from Cloudflare seeks to define value differently from the old metric of traffic so that AI gets better while content creators get paid.
“Imagine an AI engine like a block of Swiss cheese,” says Prince. “New, original content that fills one of the holes in the AI engine’s block of cheese is more valuable than repetitive, low-value content that unfortunately dominates much of the Web today.”
“We believe that if we can begin to score and value content not on how much traffic it generates, but on how much it furthers knowledge – measured by how much it fills the current holes in AI engines “Swiss cheese” – we not only will help AI engines get better faster, but also potentially facilitate a new golden age of high-value content creation,” he argues.
This is interesting for publishers because it could drastically change the type of articles, podcasts and videos produced. Could content be more focused on filling the missing parts of a knowledge base instead of mindless click-bait? We can hope.
For now, the Cloudflare model for pay-per-crawl is still in private beta. And it’s unclear if it will gain acceptance among a broad set of publishers – some may worry about being left out altogether in an AI answer.
On a positive note, some major publishers including Condé Nast and Associated Press, as well as social media companies such as Reddit and Pinterest, are supporting the Cloudflare move.
Could this lead to a fairer deal? Already, some AI companies have inked agreements with publishers, who likely do not object to their content being for AI as long as they are paid fairly.
Reddit, for example, has sued Anthropic but signed a licensing deal with Google. The New York Times, meanwhile, has sued OpenAI and Microsoft over the use of its copyrighted work.
What’s happening isn’t so different from the past. Thirty years ago, traditional news outlets were caught out when Google arrived to disintermediate them from audiences – everybody found their news and information on Google.
Eventually, publishers such as The New York Times built up their digital content and worked with the likes of Google to refer users to their websites and sell advertisements and subscriptions.
The way forward could be similar deals with today’s new AI intermediaries, which will soon find that they need good original content to deliver the answers people ask for. As before, someone has to pay for it.
This time round, though, the content creators have started asking for a better deal upfront. They have seen this movie before.
