Apple details new iPhone features like door detection and live annotations

World Accessibility Awareness Day is Thursday, so Apple started it Newsroom Blog This week to announce several major new accessibility features heading to the iPhone, Apple Watch, iPad, and Mac.

The Live Captions feature will likely be the most used, which will be available on iPhone, Mac, and iPad. The feature shows live-updated, AI-based subtitles for speech coming from any audio source on the phone, whether the user is “on a phone or FaceTime call, using video conferencing or a social media app, streaming media content, or having a conversation with someone next to them.” “.

The text (which users can resize as desired) appears at the top of the screen and points while the subject is talking. In addition, Mac users will be able to type responses and have them read aloud to others on the call. Live Captions will enter public beta on supported devices (“iPhone 11 and later, iPad models with the A12 Bionic chip and later, and Macs with Apple silicon”) later this year.

There is also a door detection feature. Unfortunately, it will only work on iPhones and iPads with a lidar sensor (such as the iPhone 12 Pro, iPhone 13 Pro, or recent iPad Pro models), but it seems useful for people who are blind or have low vision. It uses the iPhone’s camera and augmented reality sensors, along with machine learning, to locate doors and tell users aloud where the door is, whether it’s open or closed, how it can be opened, and what it might be for writing or stickers.

Door detection will join the discovery of people and photo descriptions in the new “Detection Mode” for blind and visually impaired users in iOS and iPadOS. However, the Apple blog post did not mention when this feature will be launched.

Other accessibility additions that Apple says are very close include 20 new Voice Over languages, new hand gestures on the Apple Watch, and a feature that allows players to take help from a “friend” with another game console without disconnecting from them. In addition, there are new Siri and Apple Books customizations aimed at expanding accessibility for people with disabilities, voice recognition customizations, and Apple Watch screen mirroring on the iPhone — which gives Watch users access to many of the accessibility features that are available on the iPhone but not on the watch. .

Technology enthusiasts often lament that smartphones (and personal technology in general) have become stagnant, without many exciting new developments. But this is far from the truth for many people with disabilities. Google, Apple, and many researchers and startups have made major breakthroughs, bringing powerful new access features to mobile devices.

Picture list by apple