Microsoft’s Copilot Vision Powers Edge with AI Web Page Reading, Q&A Features
Microsoft’s Copilot Vision allows AI to read web pages and answer questions in Edge. It can now interpret text and images for tasks such as recipe searching in a limited preview in the United States.
Microsoft’s AI can now read your screen or rather the websites you’re viewing. Copilot Vision is a tool that can understand and answer questions about sites you’re visiting with Microsoft Edge. The company started rolling out a limited preview of the tool on Thursday, but only in the U.S. Copilot Labs is an opt-in program for testing AI features.
Copilot Vision can look at text and images on web pages to answer questions like “What’s the recipe for this lasagna?” Keep in mind that to use Copilot Labs, you need to subscribe to Microsoft’s $20-a-month Copilot Pro plan.
Copilot Vision can do more than just answer questions. It can also summarize and translate text, as well as do things like highlight sale items in a store catalog. Additionally, it can assist you in playing games, such as providing tips during Chess.com matches.
Microsoft wrote in a blog post that TechCrunch saw, “When you choose to enable Copilot Vision, it sees the page you’re on, reads along with you, and you can talk through the problem you’re facing together.” “It’s a new way to bring AI with you as you browse the web. You can find it at the bottom of your Edge browser whenever you need help.”
Microsoft wants to make sure that AI privacy mistakes don’t get worse, so they are emphasizing that Copilot Vision deletes data at the end of each session. The company asserts that it does not save or use processed audio, images, or text for model training, at least not during this preview run.
Moreover, Copilot Vision is limited to viewing specific types of websites. For now, Microsoft is only allowing Vision to operate on a list of “popular” sites, as it does not permit it to access “sensitive” or paywalled content. What exactly does “sensitive” mean? Porn? Very violent violence? Microsoft did not say.
Microsoft is exercising caution, partly due to its involvement in legal disputes with news outlets. The New York Times claims that Microsoft allowed people to bypass its paywall by displaying NYT articles on Bing’s Copilot chatbot. This lawsuit is still going on.
Publishers Block AI Crawlers Over Data and Cost Concerns
A lot of big publishers have decided to stop AI tools from crawling their websites. They take this action due to their concerns about unauthorized use of their data and the potential for these tools to significantly increase their server costs.
Microsoft said that Copilot Vision will follow sites’ “machine-readable controls on AI,” such as rules that say bots can’t scrape data to train AI. However, the company has not specified which of the numerous existing controls Vision will adhere to.
Copilot Vision might not work on some of the best news sites on the web if things keep going the way they are. But Microsoft told publishers that it would “take feedback” to ease their minds.
The company wrote on its blog, “We’ve collaborated with third-party publishers who assist us in understanding how Vision can enhance user interaction and decision-making on their pages.” “These suggestions will help us make Vision work better with web pages.”
