Apple introduced a new feature to its camera system that automatically recognizes and transcribes text in your photos, from a phone number on a business card to a whiteboard full of notes. Live Text, as the feature is called, doesn’t require a prompt or special user work – just tap the icon and you’re good to go.
Announced by Craig Federighi on the WWDC virtual stage, Live Text will arrive on iPhones with iOS 15. He demonstrated it with a few pictures, one from a whiteboard after a meeting, and a few snapshots that had restaurant signs in the background.
Tapping the Live Text button in the lower right slightly underlined recognized text, and then swiping it could be selected and copied. In the case of the whiteboard, it collected several sets of bulleted notes and used one of the restaurant signs to grab the phone number that could be called or saved.
Certain types of text strings can also be recognized: a tracking code is recognized as such and a link to the tracking URL is provided immediately. Translation can also be done quickly to or from any language supported by Apple’s other translation tools.
The feature is reminiscent of many that can be found in Google’s long-developed Lens app, and the Pixel 4 added a more robust scanning feature in 2019. The difference is that the text in every photo taken by an iPhone with the new system is captured more or less passively – you don’t have to switch to scanner mode or start a separate app.
This is a nice thing for everyone, but it could be especially helpful for people with visual impairments. A snapshot or two of any text that would otherwise be difficult to read is dictated or saved.
The process takes place entirely over the phone, so don’t worry about this information being sent to a data center. That also means it’s pretty quick, although we can’t tell if it happens right away or, as with some other machine learning features, in the next few seconds or minutes after it’s recorded. However, your back catalog of photos will be tagged with live text in the idle moments of your phone.