Microsoft and Reddit are offering conflicting explanations for why Microsoft’s search engine, Bing, is currently blocked from crawling Reddit and offering links from the site in its search results.
Reddit, which now demands payment from anyone crawling the site and using its data to train AI products, claims that Bing’s crawler is being used to power AI products. Microsoft claims it has made it easy for any site to block its crawler that’s used for AI products, while still allowing a crawler that is only used for search results, and that Reddit’s decision to block Bing is “impacting competition” in the search engine space.
The conflicting reasonings behind the block are further proof that the massive, indiscriminate scraping of the internet to create AI training data in a way that violates long-respected norms about how to access information on the web are eroding trust, making the internet less open, and causing tech companies to beef about this issue in public.