The magic indicator is active Googlebook Built with Google DeepMind. The research team behind this key capability has shared more information about AI-powered indicators.
DeepMind wants to use AI “to help the pointer understand not only what it’s showing, but why it’s important to the user.”
Our goal is to overcome a common frustration: since a typical AI tool lives in its own window, users have to draw their world into it. We want the opposite: intuitive AI that meets users across all the tools they use without interrupting their flow. For example, imagine asking “Show me directions” by pointing to a picture of a building. When the AI system already understands the context, nothing more is needed.
The idea is to replace “text-heavy prompts with simpler, more intuitive interactions.”
An AI-powered pointer will facilitate this process by fluidly keeping the visual and semantic context around the pointer, allowing the computer to “see” and understand what is important to the user.
Similarly, “This AI system, which understands a combination of context, markup and speech, will allow users to make complex queries with natural shorthand.”
Example use cases include:
- Can point to a PDF and request a bulleted summary to paste directly into an email
- hover over the statistics table and request the pie chart version
- highlight a recipe and ask to double all the ingredients
In the example below, “a freeze frame in a travel video becomes a booking link for that nice-looking restaurant.”
Google’s AI Studio has two AI-enabled indicator demos:
Additionally, you’ll soon be able to “ask Gemini in Chrome about a part of a web page you’re interested in using your browser.” It is in the process of being rolled out.
For example, you can choose several products on one page and ask them to compare, or show where you want to visualize a new sofa in your living room.
FTC: We use automatic affiliate links that generate income. More.
