Reddit has taken a stand against AI crawlers

Reddit changes its robots.txt file to prevent AI companies from using its content without payment. This is part of a larger trend of content owners and AI companies fighting over how to use data.

Reddit is against AI companies or wants them to pay up. Reddit said earlier this week that it would be changing its robots.txt file, which is also known as its Robots Exclusion Protocol.

Companies that own the content are either negotiating with or fighting with AI companies that want to use the content to train their language models. This dry-sounding edit is part of that fight.

Websites use “robots.txt” to instruct other websites on how to crawl them. A classic example of this is when websites allow Google to crawl them, enabling their inclusion in search results.

Protecting Content and Traffic Exchange

When it comes to AI, the value exchange is not as clear. If you run a website that depends on getting clicks and visitors, it doesn’t make sense to let AI companies take your content and then not send you any traffic or, worse, copy your work.

Reddit appears to be attempting to prevent the kinds of actions that businesses like Perplexity AI have come under fire for, that it changed its robots.txt file and continues to limit and block bots and crawlers that it doesn’t know about.