AI search engines send 96% less traffic to news sites compared to Google search: Report

A new report by content licensing platform TollBit, shared exclusively with Forbes, has revealed a significant drop in referral traffic from AI-powered search engines to news websites and publishers. The report found that AI-driven search tools, such as those developed by OpenAI and Perplexity, send 96% less traffic to publishers compared to traditional Google search.

Despite claims by AI firms that their technology will drive new revenue opportunities for publishers, the data suggests otherwise. AI developers have doubled their web scraping activities in recent months, with companies like OpenAI, Perplexity, and Meta collectively scraping websites an average of two million times in the fourth quarter of 2024 alone. The report, which analysed 160 websites across national and local news, consumer tech, and shopping blogs, found that each page was scraped about seven times on average.

Toshit Panigrahi, CEO of TollBit, highlighted the scale of this extraction. "We are seeing an influx of bots that are hammering these sites every time a user asks a question," he told Forbes. "The amount of demand for publisher content is nontrivial."

How AI Search Engines Impact Publishers

Nathan Schultz, CEO of Edtech company Chegg, has been vocal about the risks AI search engines pose to businesses that rely on search traffic. :It’s time to say no," Schultz told Forbes, arguing that the long-standing balance between content creators and search engines is being disrupted.

Chegg’s experience serves as a warning. In January 2025, its website traffic plummeted by 49% year-over-year, a sharp contrast to the 8% drop in Q2 2024, just after Google introduced AI summaries. The decline has been so severe that Chegg is now considering going private or seeking acquisition.

TollBit’s report found that AI companies scraped websites an average of 2 million times per quarter in 2024. However, these scrapings rarely translate into traffic for publishers. Each page was scraped an average of seven times, adding to server costs without generating additional revenue.

Opaque Scraping Practices

One of the major challenges for publishers is the lack of transparency regarding AI web crawlers. AI firms use various user agents to collect data, but many do not properly disclose their activities. Olivia Joslin, co-founder of TollBit, pointed out the dilemma faced by publishers. "It’s very hard for publishers to want to block Google. It could impact their SEO, and it's impossible for us to deduce what exactly their bots’ use case is for," she said to Forbes.

Perplexity, valued at $9 billion, has come under scrutiny for its scraping methods. Even when publishers attempt to block its access, the company continues to send referral traffic, suggesting undisclosed scraping techniques. The report found that Perplexity scraped a particular publisher’s site 500 times while sending over 10,000 referrals, raising suspicions about the methods used to access the content.

The AI firm previously faced backlash for republishing paywalled articles from major outlets like Forbes, CNBC, and Bloomberg without adequate attribution. It also surfaced AI-generated blogs and low-quality content in search results. CEO Aravind Srinivas admitted to issues with its republishing feature, calling it a work in progress.

Finding a Way Forward

Some publishers have turned to licensing deals with AI companies to ensure fair compensation. For example, The Associated Press, Axel Springer, and The Financial Times have negotiated content agreements with OpenAI. Others, like TollBit, have implemented models that charge AI companies per scraping instance.

If left unchecked, analysts warn of an impending "AI slurry", a scenario where high-quality journalism is undermined, forcing trusted content creators out of business and flooding the internet with lower-quality information.

The Future of AI and Search

AI-powered search engines are not going away. As more users turn to AI chatbots and research tools instead of traditional search engines, the industry must find ways to balance innovation with fair compensation for content creators. Without clear frameworks and enforceable licensing models, the sustainability of digital journalism and the quality of information online remains at risk.