Supposedly, it is not supposed to scraping websites

Websites Extended by Web browsers are allegedly restrictions according to New CloudFlare report. Specifically, when the company’s boots seem to be “stealth crawling”, “robots.Txt moving files and firewalls.

Robots.txt Web browsers can violate the content of the websites. Official Perplexity Web browsing boots They are “perplexitybot” and “perplexity-user”. In the tests of CloudFlare, unease was still able to display the content of a new and new website, even when the boots blocked Robots.txt. Web browsers spread to websites with web pages (WAF).

CloudFlare believes that the generic browser for the impersonation of Google Chrome Macos “Robots.txt prohibits normal boots. In Cloudlfare, the company’s cone-free browser could not be listed in the official IP of the Permide. CloudFlare He says it is doing complexity with the numbers of the autonomous systems. It is an IP address identifier for the business operating.

The Report of Engadget Cloudflare has arrived in a uneasiness. We will update this article if we return.

Up-to-date website information is essential for companies who train models, especially since the service is used as search engine substitutions. It has also been captured in the past, avoiding the rules of being up to date. In 2024 several websites reported Although uneasiness was still banned from Robots.txt, the company was accused of third-party web browsers who thought he was using it at the time. Afterwards He met with many publishers To share the income earned from the content displayed together, it looks good for his past behavior.

The network of scraping networks will be a whack-a-mole game stop. Meanwhile, CloudFlare has removed his boots Verified Boot List And he set a way to identify and block the way to access the content of its customers.