- Expressive Captions add feelings and [gasp] emotions to Android’s Live Captions
- Google Lookout gets new Gemini enhancements
- Gemini on Pixel phones will now remember more information about you and your interests
Google has released a number of new AI features for most Android phones, plus a couple that work only on its Pixel range of devices.
Some of the new features work in Google Gemini, Google’s AI app, but most of them work throughout the whole Android operating system, which means there will be some big changes for Android users. Let’s break it down.
Expressive Captions
First up are Expressive Captions, which are a way of bringing more emotions into the rather dull and lifeless world of subtitles and captions. Google introduced Live Captions to Android back in 2019, so if anything plays audio on your phone, it also has captions. They work much the same way that automatic subtitles do on YouTube videos – your phone generates them live, based on what it interprets the person is saying.
Live Captions, however, remain rather lifeless. With Expressive Captions, Google is utilizing the power of AI to help you get more of the context of what’s happening. You’ll see comments in square brackets like [whispers] and [cheers and applause] appearing amongst captions, giving you a better understanding of the feelings and emotions being conveyed.
Intense vocalizations will be written in [ALL CAPS], while vocalizations like [sighing], [grunting] and [gasping] will also appear. Ambient background sounds will also be tagged. Because Live Captions are part of the operating system you’ll now get Expressive Captions with most things you watch, so look out for them.
Gemini in Lookout
Google Lookout is an assisted-vision app that’s available on Android devices via the Play Store. It uses AI to help people with low levels of vision, or sight loss, explore the world around them using their phone’s camera. You can use it for reading out text, scanning food labels, currency identification, and even scanning your surroundings to identify things like doors and windows and find out how far away they are.
Now Gemini AI is part of the Lookout app, which means it has much richer, more helpful descriptions. If you take a photo in the app you’ll now hear its description read aloud in a natural-sounding voice, and you can ask Loopout follow-up questions about it.
New Gemini extensions
The more extensions that Google Gemini has, the more apps it can bring its AI magic to. Now Gemini has a new Spotify extension, which means you can play your favorite songs or discover playlists from within Gemini.
Gemini will now also connect to your phone and messages app, meaning you can call contacts and send messages from within Gemini. Google says that “soon” you’ll be able to use Gemini to control your smart home devices. Finally, Gemini will now also connect to Maps, which means you’ll get more information about places.
More Android features
Also in the December update you’ll find new pizza-theme sticker combos that you can share with friends using Emoji Kitchen in Gboard, Google’s keyboard app. You can also use QR codes in Quick Share, which makes it easy to share pictures, videos and documents without having to add people as contacts or verify devices. Now you just generate a QR code for people to scan. Documents you scan using your phone will also now be improved, with better contrast and white balance.
Pixel-only features
If you’ve got a Google Pixel phone then you get some extra features in the December Android update. There are a lot of small updates to things like the camera and security settings, but the most important feature is Gemini Saved Info. This means you can ask Gemini to remember more about you, what you’re interested in, and what you like, so that it can tailor its responses to more of what you’ll be interested in.
Pixel users will also get some updates to the Pixel-exclusive app, Pixel Screenshots. Pixel Screenshots uses Gemini to analyze and recall the information embedded in your screenshots. Now you can use Pixel Screenshots to save things when you’re using Circle to Search with just one tap.