Meta Builds Search Index for AI Chatbot, Shifts Away from Google News

Meta is building a search index for its AI chatbot so that it doesn’t have to rely on Google for short summaries of current events.

Meta is reportedly building a search engine index for its AI chatbot so that it doesn’t have to rely on Google for summaries of current events created by AI. Meta AI looks like it’s moving on to the next level, which is becoming a fully independent AI search engine.

Since at least the summer of 2016, Meta has been using a user agent named Meta-External Agent to crawl the web. Many reports of excessive crawling have surfaced in various forums. For example, on Hacker News, someone said they got 50,000 hits from the bot.

One Webmaster World bot crawling forum post says that even though Meta-ExternalAgent’s documentation says it respects robots.txt, the bot never went to that file, so it wouldn’t have made a difference.

The bot might not have been fully ready at the beginning of the year, but its undesirable behavior has since calmed down. The bot summarizes search results, and its results show that its goal is to make people less dependent on Google and Bing for search results.

Meta AI’s Potential Challenge to Google in AI Search

This may be the start of a challenge to Google and other search engines in AI search. From what we currently know, it appears that they are developing a search index to complement their meta AI.

The Verge reports that Meta is searching websites for search summaries to incorporate into the Meta AI Chatbot.

Meta AI is still utilizing Google’s search index, as it bears a resemblance to a search engine. A search for the most recent Game 4 of the World Series on Meta AI yielded a summary, a correct answer, and a link to Google.