AI for the visually impaired: where visual scene description breaks
The model confidently names objects — "person, table, laptop" — and that is where the visual scene description ends. A blind user needs not a list but relationships: who is holding the document, where the arrow on the diagram points, what changed between two images.
As soon as a scene cannot be reduced to a list of objects, the alternative description becomes formally correct and useless.
In interfaces, the gap is more noticeable than in photographs. An icon without a text label, a button state conveyed only by color, a tooltip that disappears after a couple of seconds — these are interface inaccessibility in its purest form.
A screen reader traverses the markup in its own order, and if it does not match the visual order, the user rebuilds the picture from scratch, guessing which field the error message belongs to.
Recognition errors here are more dangerous than silence. A confident caption where the meaning is read incorrectly leaves no chance to doubt: the user cannot see the image to verify.
Hard cases are predictable — dense tables, handwritten text, small charts, scenes with several people and the words "he," "she," "left."
The cost of these failures is shared between the user and support service.
The former cannot complete a payment or book an appointment and does not understand at which step they are stuck; the latter receives requests without technical specifics, while the team manually corrects descriptions after every update. Regression returns with any layout change as long as checks do not cover key scenarios.
We examine how this is addressed: where to set the confidence threshold and what to do when it is not reached, which scene types to describe by template and which to send for manual review, how to link the description to traversal order and catch regressions before release.
Autonomy, coverage, and offloading maintenance: measurable effect
The user opens a document and immediately receives a description — without contacting support or waiting for someone to manually review the file.
The system reads the structure: headings, block order, tables, chart captions — and assembles coherent text instead of a set of disconnected hints. The task is resolved at the moment the material is accessed.
Coverage of content types is defined by policy, not one-off improvements. Presentations, reports, interface screens, scans, and diagrams follow a single processing path, while alternative text requirements are checked against WCAG — the depth of description becomes predictable.
If the source is unreadable, the system reports the limitation directly, and it does not turn into a hidden defect.
For maintainers, the composition of work changes. Reviewing similar files goes to automation; people handle disputed cases: complex diagrams, narrow terminology, edits to the description policy itself.
The reduction in support requests comes not from quickly closed tickets, but because most requests simply do not arise.
Product accessibility becomes a property of the release, not a separate project tied to a date. A new content type is connected by changing settings, and the description rule lives in the shared build and is reused by all teams.
Operating modes: streaming description, on demand, hybrid pipeline
The operating mode determines how frames enter processing: as a continuous stream, as individual requests, or mixed.
This affects the delay between an event in the frame and its description, the load on the compute pipeline, and how the system behaves when frames are lost or the connection drops.
Streaming description feeds frames for recognition at a fixed interval and maintains a constant queue — the description arrives almost immediately, and the pipeline load is even.
On-demand description starts processing on a user action or external event: resources are spent only on the needed segments, but everything that happened between requests remains undescribed.
The hybrid pipeline combines branches: one is responsible for continuity, the other for detailed analysis on demand.
| Mode | When it fits | Limitations |
|---|---|---|
| Streaming description | Live broadcasts, real-time events, scenes with rapid shot changes | Constant load; with a narrow channel, the queue grows and the description lags behind the image |
| On-demand description | Archives, recordings, rare access to material | Events missed between requests; delay on the first response |
| Hybrid pipeline | Services that need both live description and on-demand analysis | More difficult to maintain: priority rules and queue separation are designed in advance |
Each mode breaks in its own way. Streaming hits the channel bandwidth and queue depth, on-demand mode hits the rarity of requests, while hybrid more often suffers from priority conflicts when on-demand analysis displaces the live stream.
Implementation stages: from scenario audit to handover of policies
The work order is structured so that an error at an early stage does not turn into rework at a late stage: first we analyze what to describe and why, then we design the flow — and only after that do we configure the generation of wording.
- Scenario and content audit — inventory of image types and their frequency, identification of cases where description is critical for understanding and where a short caption is sufficient. The output is a list of priority scenarios and markup rules.
- Processing scheme design — how an image enters the system, where it is stored, how it is queued, what happens on failure. Separately, we record the points where a human editor is needed.
- Description setup — volume and style of text for each content class, dictionaries of terms and names, a unified tone. We test templates on material the system has not seen yet.
- User testing — blind and visually impaired participants listen to descriptions in a screen reader; we note where the wording is confusing or delayed. We make edits according to protocol, not taste.
- Testing on the production stream — share of images with a correct description, time from upload to finished text, share of cases sent for manual revision.
- Handover of policies — instructions for the content team, escalation procedure, rules for updating dictionaries and re-checking after a source change.
Each stage is closed with an artifact: without it, the next one does not begin, so the result is reproducible by your team.
Handover package: models, annotations, description dictionaries, policies
After delivery, the client retains not only a working service but the entire package by which the system can be maintained in-house or handed over to another team without reverse engineering. Below is what exactly is included in the handover and why each element is needed.
- Configurations and deployment schemes — environments, connection parameters to frame storage and the processing queue, the procedure for rolling out model updates without stopping the service.
- Description dictionary — reference wording for scene classes with rules: where the description is built from a template, and where the model formulates freely and within what boundaries that is acceptable.
- Content annotation — a corpus of annotated frames and video fragments, annotation agreement, and inter-annotator agreement metric.
- Frame retention policies — source lifetimes, masking of faces and personal data, what goes into the training pipeline and what is deleted immediately after processing.
- System documentation — architecture diagram, interface contracts, instructions for connecting a new video source without retraining.
- Maintenance policies — procedure for updating the dictionary and annotation, criteria for degradation of description quality, rollback plan to the previous model set.
- Access matrix — roles for annotation, frame export, and editing the description dictionary.
The package removes the main risk — dependence on a single contractor: the dictionary and annotation can be expanded within the team, and disputed scenes can be analyzed according to the agreement without rebuilding the corpus from scratch.
Case study: describing product displays and interfaces for blind employees
What was there before implementation
The client is a retail chain where some employees in the sales floor and warehouse have visual impairments. Internal systems — goods receiving, order cards, inventory — were built as dense tables: icons without labels, statuses conveyed by color, rows not separated by headings.
In NVDA, such screens were read as a stream of unnamed buttons, and the employee did not understand where one item ended and the next began.
Metrics before and after
Before implementation, 3 out of 10 operations were completed without an assistant; the rest had to be handed over to a colleague. One item receiving took 6–8 minutes: the number and quantity had to be asked again verbally.
We connected the model to the stream of images and product data: photos of the shelf, packaging, and invoice are turned into a short description — name, quantity, packaging condition, position on the shelf.
Interface descriptions are added to screen markup: element labels, verbal statuses instead of color ones. The description length was limited so that speech output would not "choke" and the employee would have time to complete the operation.
After: 9 out of 10 operations without an assistant, item receiving takes about 2 minutes, and the share of errors due to an incorrectly read invoice decreased threefold.
We repeat measurements after every markup change: accessibility regression returns quietly, and it is visible only in operation completion time.
How does local frame processing differ from cloud processing?
The answer depends on where frame recognition physically happens. With local processing, the model runs on the device itself — the frame does not leave the device and is turned into a description on the spot.
With cloud processing, the frame goes to a server where a heavier model runs, and the finished text is returned back over the network.
| Option | When it fits | Limitations |
|---|---|---|
| Local processing | Frame privacy and offline operation are critical | More compact model: shorter description, handles complex scenes worse |
| Cloud processing | Detailed descriptions are needed and there is a stable channel | The frame leaves the device, the system does not work without a connection, response depends on the channel |
| Hybrid scheme | A balance is needed: basic hints immediately, details on demand | Two models and switching rules — harder to maintain and debug |
Frame privacy is addressed at the architecture level: if the image does not leave the device, the client does not acquire obligations to store and protect the video stream, and the user does not need an explanation of where their frames go. Local mode also resolves the consent issue — processing remains within the device itself.
System response in local mode is predictable: it does not depend on connection quality or cloud load, so a hint arrives equally quickly both in the subway and in dense urban areas.
Offline operation is possible only with a local model — cloud is unavailable in that case, and hybrid switches to a limited mode and describes only what it manages to process on the spot.
We do not lock in one mode forever: basic recognition runs on the device, while complex scenes go to the cloud on an explicit user request and when a channel is available.
Frame privacy remains manageable, response is predictable, and the system does not fail when there is no connection.
Technical concerns: noise, low light, rare content formats
The AI system describes not staged videos but real recordings: phone footage indoors, evening light, moving objects, background noise.
That is why we check description quality not on a demo example but on a sample from the client's archive — it shows where recognition falls short and which scenes require a separate mode.
We track two indicators separately: object recognition accuracy and suitability of the description for reading aloud. The second is more important — an incorrectly named object confuses the listener, while an extra detail stretches the voiceover.
Therefore only significant scene elements go into the text, and doubtful fragments are either clarified or replaced with a generalized wording.
What if recordings are shot in low light and with noise?
Frames undergo brightness equalization and noise reduction; the description is built on stable features: contours of large objects, their relative position, movement in the frame.
Small details that the system cannot distinguish confidently do not go into the text — instead, a generalized scene description is given.
Are rare formats supported — panoramas, still frames, diagrams, infographics?
Each format has its own mode: a panorama is described sequentially, infographics — with reading captions and reconstructing connections between blocks, a still frame receives an extended description without being tied to movement.
If confidence is below the threshold, the material goes to an operator for review rather than to voiceover.
How robust is recognition on our content, and can the trigger threshold be configured?
The threshold is configured by content category and checked on an annotated sample from your archive: we look at the share of missed significant objects and the share of false positives. Describe the task — we will come back with an estimate and a validation plan on your materials.
Implementing AI for the visually impaired: let's discuss scenarios and limitations
We start the conversation not with a list of technologies but with an analysis of the scenario: where exactly visual content remains inaccessible and what prevents closing this now.
To estimate the task, what is already in progress is enough — there is no need to prepare documents in advance.
To come back with a work plan rather than general words, send:
- content examples: 15-20 typical images and short videos, including complex ones — screenshots of interfaces, diagrams, charts, frames with embedded text;
- publication flow: where the material comes from (editorial system, application, archive) and where the finished description goes;
- accessibility requirements: the conformance level to be achieved and the rules already adopted within the company;
- output points: where the description should appear — screen reader software, mobile app, search index, voice assistant;
- limitations: languages, sensitive topics, prohibition on mentioning persons, description retention periods;
- approval side: who accepts the result and at which stage.
Then we come back with a task estimate and plan: breakdown into stages, what we check at each one, where your input is needed to approve the stages.
Before the start, we show the prototype working on your materials — you can see how the system describes complex frames and where it makes mistakes, even before AI implementation begins in the main product.
