Reddit ($RDDT) Updates Robots.txt to Safeguard Against AI Crawlers

Jun 25, 2024, 07:14 PM

Reddit has announced upcoming changes to its platform aimed at safeguarding against AI crawlers. The company plans to update its robots.txt file to clearly specify that companies using automated agents to access Reddit must adhere to its terms. This move is part of Reddit's broader strategy to block and rate-limit unknown bots and crawlers. Additionally, Reddit will continue to allow good faith actors, such as researchers and organizations like the Internet Archive, to access its content for non-commercial use. These measures come as companies like Google and OpenAI increasingly license Reddit data to train large language models, which has led to concerns about the quality and authenticity of the content being used. The announcement was highlighted by TechCrunch and Reddit's stock symbol is $RDDT.