Reddit CEO Says Microsoft is Scraping Reddit’s Data without Permission

In a recent interview, Reddit CEO Steve Huffman called out Microsoft and others to pay if they still want to continue scrapping Reddit’s data.   

So far there’s only Google and Open AI, who have struck a deal to pay Reddit for the rights to access its data. This is why Huffman in an interview this week specifically mentioned Microsoft, Anthropic, and Perplexity for refusing to negotiate in regard to using Reddit’s data. Saying that “It has been a real pain in the ass to block these companies”

Huffman also stated that “Without these agreements, we don’t have any say or knowledge of how our data is displayed and what it’s used for, which has put us in a position now of blocking folks who haven’t been willing to come to terms with how we’d like our data to be used or not used”

In the last couple of years Google accounted for almost 40% to 50% of Reddit’s traffic in a single day, Huffman said. 

And, this is probably why Reddit used to allow search engines to access the site’s data for free — That’s all because of how much traffic is coming back to Reddit. 

“When it was used for simple search, to create simple links that would send us traffic from search engines, that was fine. But now folks are using Reddit data for training, they’re reselling it, they’re doing search summaries instead of linking to us” — Huffman said. 

And, What Did Microsoft Replied… 

In an official X post, Microsoft’s head of search, Jordi Ribas said that “the company has provided all publishers including Reddit with webmaster crawling controls in September 2023”

Another spokesperson of Microsoft said that “the company honors the directions provided by websites that do not want content on their pages to be used with our generative AI models.”

And, that bing has stopped crawling on Reddit on July 1 after they implemented their updated robots.txt file. Which basically was all about prohibiting crawling on Reddit without paying the company.