Non-Google search engines blocked from showing recent Reddit results

Recent updates to Reddit's Content Policy have led to non-Google search engines being blocked from displaying recent Reddit results. This change stems from Reddit's new robots.txt file, which prohibits crawling its site without an agreement that aligns with its rules, particularly regarding the use of Reddit content for AI training. As a result, search engines like Bing, DuckDuckGo, and Mojeek are showing minimal or no recent Reddit results, while Google remains the only search engine able to access this content. Reddit's decision was influenced by an increase in commercial entities scraping its content without adhering to its terms. Although Reddit has expressed willingness to collaborate with "good-faith actors," many search engines have struggled to reach agreements. Reddit has signed a lucrative deal with Google, reportedly worth $60 million annually, allowing Google to utilize Reddit data for AI training. Meanwhile, Microsoft has confirmed that Bing ceased crawling Reddit following the updated robots.txt file. The implications of this change may hinder competition among search engines, as they rely heavily on access to content from platforms like Reddit. Concerns have been raised about the potential negative impact on smaller websites, as the trend of restricting access could limit their traffic sources. Reddit's spokesperson emphasized the need for constraints on for-profit companies that scrape content for AI purposes, highlighting the broader implications for the health of the Internet.