FB16265340: Vision should expose the image description generation used by VoiceOver

[samhenrigold]

• Date: 2025-01-08
• Resolution: Open
• Area: Vision
• OS: iOS
• Type: Suggestion
• Keywords: CoreSceneUnderstanding, VoiceOver, accessibility, alt text

Description

In a few areas throughout the system, rich captions/descriptions are generated for images:

• When texts with images are announced through AirPods
• In the Magnifier app in Detect mode
• In VoiceOver with “Image Descriptions” enabled

The image attached results in a description like “A white chair next to a wooden table on a white rug” while the current Vision framework might only be capable of picking out a chair, a table. An API for these richer descriptions would allow app developers to make more accessible apps.

As a real example, I work on the Mastodon social app. It would be a huge benefit to pre-populate some placeholder alt text for images when composing a new post.

Files