AI startup Perplexity is being accused of scraping content from websites that have explicitly prohibited such actions, according to research published by internet infrastructure provider Cloudflare. On Monday, Cloudflare detailed its observations of Perplexity’s activities, noting that the startup appears to have obscured its identity to bypass website preferences regarding crawling and scraping.
The effectiveness of AI products like those developed by Perplexity often relies heavily on vast amounts of online data. In response, many websites have implemented measures such as the Robots.txt file, which indicates which pages can be indexed. However, the results of these efforts have been mixed. Cloudflare’s findings suggest that Perplexity has been circumventing these blocks by modifying its bots’ “user agent” and altering its network identifiers to disguise its crawling activities.
Reports from Cloudflare indicate that they detected this scraping behavior across numerous domains, amounting to millions of requests daily. The company utilized machine learning and network signals to verify the activity of Perplexity’s crawlers. Cloudflare noted that clients had lodged complaints regarding Perplexity’s actions even after implementing rules to block its known bots.
In response to these allegations, Perplexity spokesperson Jesse Dwyer characterized the Cloudflare report as a “sales pitch,” asserting in an email that the documented screenshots indicated no content had been accessed. Dwyer also claimed that the bot identified in the Cloudflare report does not belong to their company.
Cloudflare has recently intensified its opposition against AI crawlers. In addition to addressing concerns about Perplexity, they launched a marketplace that allows website owners to charge AI scrapers and introduced tools aimed at preventing unauthorized scraping. This controversy is not the first for Perplexity, as the company has faced similar accusations regarding plagiarism and ethical concerns surrounding its data usage.
Source: https://techcrunch.com/2025/08/04/perplexity-accused-of-scraping-websites-that-explicitly-blocked-ai-scraping/