Reddit will improve a web standard used on its platform to prevent automated data scraping. The social media site is responding to reports of Artificial intelligence startups bypassing the rule to gather content for their systems.
The Robots Exclusion Protocol, or “robots.txt,” will be updated on Reddit. It is a commonly recognized norm. What portions of a website can be crawled is determined by the Protocol. This action is being taken in response to claims made by publishers that Generative AI businesses are copying their work.
The company also announced that it will continue employing rate-limiting. A method used to control the number of requests from any single entity. Additionally, Reddit plans to block unknown bots and crawlers from scraping data—collecting and saving raw information—from its site.
Recently, robots.txt has become an essential tool for publishers to prevent tech companies from using their content without charge to train AI algorithms and create summaries in response to search queries.
A letter from the content licensing startup TollBit to publishers last week highlighted that several AI firms were evading the web standard to scrape content from publisher sites. This follows a Wired investigation which discovered that the AI search startup Perplexity likely circumvented efforts to block its web crawler via robots.txt.
Earlier in June, Forbes accused Perplexity of plagiarizing its investigative stories for use in generative AI systems without proper attribution.
Reddit stated that researchers and organizations, such as the Internet Archive, would continue to have access to its content for non-commercial purposes.
.
