RPA bots with computer vision: where conventional automation breaks
Some systems in the customer's environment do not expose data programmatically: there is no API, no integration gateway is provided, and the client itself is a terminal window or a thin web front end with closed authorization.
This is exactly where automation hits a bottleneck: a person manually transfers data between the accounting system, email, and spreadsheets, because no other access to the interface exists.
The cost of this bottleneck grows unnoticed.
The operator keeps the sequence of windows in mind, copies details, verifies statuses — and as soon as the flow of requests increases, they become the limiter of the entire process: the queue builds up, transfer errors surface at later stages, and vacation or sick leave stops the data exchange.
A classic bot breaks here predictably.
It is tied to page markup or button coordinates, so failures on form updates are a standard scenario: a block shifts, the order of fields changes, a confirmation step is added — and the robot either crashes or writes data into the wrong field.
Rewriting the scenario for every interface change is more expensive than leaving manual transfer in place.
RPA bots with computer vision remove this limitation: the robot recognizes fields, buttons, and tables from the screen image and works with them just like a person, requiring neither an API nor access to internal markup. Let's look at how this is handled in practice.
The effect of implementation: speed, resilience, and relieving people
The processing speed of a single operation stops depending on human pace. The robot runs the scenario end to end — finds the window, recognizes fields, fills them in, confirms — and spends seconds on it instead of the minutes an operator needs.
More importantly, on the thousandth iteration it works the same as on the first, without a fatigue slump by the end of the shift.
The share of operations without human involvement usually stays at 70–85%: everything that fits a rule — standard invoices, requests, reconciliations — runs fully automatically.
The remainder goes into an exception queue, where the operator handles only nonstandard cases, not the entire flow. Employee workload on routine tasks drops: people move to disputed operations and customer-facing work.
At the same time, an entire class of failures disappears — input errors.
A mixed-up column, a typo in a detail, a value in the wrong field: vision reads data the same way a person sees it, and validation against rules and a reference data set cuts off incorrect data before it is written to the system. What used to be caught only during reconciliation simply does not occur.
Recovery after a failure is the sore spot of bots built on rigid scripts: a button shifts, the interface updates, the session drops.
We build scenarios with state save points: after a crash, the robot repeats the step from the last confirmed operation rather than starting the cycle over, and leaves a screenshot for investigation.
We measure recovery time in minutes instead of hours of manual reassembly — downtime is counted in time, not in team overtime. The same resilience principle is built into mature platforms such as UiPath.
Robot operating modes: from screen captures to hybrid schemes
The robot's operating mode is chosen not by preference, but by how much access the target system provides. If it has an open programmatic interface, we go through it: this is the fastest and most predictable path.
If the system is closed and exists only as an application window, screen form recognition and emulation of user actions remain.
The key selection criterion is resilience to layout changes. Coordinate binding falls apart with any update, element-name binding falls apart with renaming, while recognition by visual features survives a button moving.
Therefore, in real environments, a hybrid automation scenario almost always wins: some steps run on data, others on the image.
| Mode | When it fits | Limitations |
|---|---|---|
| Programmatic interface | The system has open data exchange points | Some operations still run only through the window |
| Working by window elements | The layout is stable and rarely changes | Breaks when markup is updated or the client version changes |
| Recognition from screen captures | The system is fully closed, with no internal data | Requires reference screen profiles and stable scale |
| Hybrid scenario | Some steps are open, others are visual only | Harder to maintain: both recognition rules and exchange logic are required |
We separately handle nonstandard windows — modal dialogs, warnings, file selection windows. Their own recognition rules and wait times are written for them; otherwise the robot will stop at the first pop-up window that was not in the recorded scenario.
How we implement a robot: stages of engineering work
A robot breaks not on complex logic, but on a rare process branch that was missed at the start. Therefore, the work goes from the employee's real actions to the scenario: first we record what happens in the interface, then we write the automation and put it into operation.
- Process discovery. We break down the employee's steps in order: where the data comes from, which systems we work in, which operations repeat, and which require human decisions. The output is a list of steps with frequency and an assessment of what we will fully automate and what we will leave for approval.
- Map of screens and transitions. We record every element the robot must find, loading states, pop-up windows, and the order of transitions between forms. The map immediately shows where the scenario will hit an interface change.
- Configuration of the robot scenario. We build logic around finding elements by visual features, not by screen coordinates: when blocks shift, the robot keeps working instead of clicking into empty space.
- Exception testing. We run the scenario through nonstandard situations — an empty selection, a slow system response, a changed form, an expired session. Each case gets its own branch and log entry so investigation takes minutes.
- Test run. We compare the robot's result with the employee's work on accumulated data and look at discrepancies: they show which validations are missing.
- Handover to operation. We provide documentation and a regulation for startup and failure investigation, and train those who will manage and modify the scenarios without us.
Handover package: diagrams, regulations, operator instructions
After delivery, the customer retains not only a working robot but also a package that their team can use to deploy scenarios, change access rights, and investigate failures without our involvement.
- Robot scenario diagram — the steps of operation processing with branching points, stop conditions, and places where the accounting system is accessed. It shows which failure stops the process and which sends the robot into a retry.
- Working scenario configuration — recognition parameters, screen form templates, and confidence thresholds moved outside. The threshold is adjusted without changing code.
- Startup and restart regulation — the startup procedure, actions for a frozen window, and rules for continuing an interrupted operation so that duplicates do not appear.
- Account access rules — what rights the robot needs, how a separate service account is created, and how access is revoked when a password changes or an employee leaves.
- Operator instruction — decoding of statuses, actions for a typical recognition error, and the moment when support should be called. Intended for an employee with no development experience.
- Operation processing log — for each operation, the time, result, and screenshot of a disputed step. This is both evidence for audit and material for tuning thresholds.
- Dependency map — which screen forms and interfaces of accounting systems the robot touches and what breaks when they are updated.
The package is assembled so that typical tasks — password changes, restart after a failure, investigation of a frozen operation — are handled by the customer's team themselves. Support remains, but not as the only entry point for any change.
Case study: request processing in a system without programmatic access
A building materials distributor received requests by email and through a form on the website, and operators manually transferred them into the 1С accounting system without a published programmatic interface.
One request had 12–14 fields: counterparty, item, volume, delivery address, stock balance and price reconciliation.
By the start of the shift, 400–600 unprocessed requests remained in the queue: emails received overnight were waiting for an operator, and response time to an inquiry reached 6 hours.
Three operators were busy with repetitive data entry, and after promotions the schedule broke down: some requests slipped to the next day.
Request processing by an RPA bot with computer vision was built as follows: the agent works in the same program window as a person, recognizes the form, fields, and rows of the tabular section from the image, inserts data from the email, and confirms the operation.
In case of a mismatch — a new counterparty, an unfamiliar item, a price discrepancy — the request is flagged and goes to an operator.
The request queue before and after is incomparable: by the start of the shift, 40–50 items remain, and response time to an inquiry has dropped to 10–15 minutes. The robot closes about 900 requests per day, and the share of exceptions requiring a human is 6–8%. Two operators moved to discrepancy reconciliation and work with new counterparties.
The before-and-after metrics do not rely on typing speed: a log is written for each operation — what was recognized, what was recorded, and at which step the agent stopped. The same environment is later transferred to shipment confirmations, acts, and returns.
What to do if the target system updates its interface?
The scenario anchors not to button coordinates but to stable element features: caption text, role in the accessibility tree, icon, neighboring labels, relative position in the block.
An interface update in the target system — moving a button to another corner, changing color, rearranging fields — preserves such features, and the scenario continues working without intervention.
When the rework is deeper: the caption wording changed, a confirmation step was added, the form was split into two screens — the bot searches for the element by a combination of features and, if recognition confidence drops, does not click at random but records the discrepancy in the log.
Correcting the scenario after an update comes down to a few actions: update the area reference, add a new feature to the step description, and add a branch for the window that appeared.
We keep scenario health under control with regular test-environment runs: the bot follows the route, the result is compared with the expected one, and if there is a deviation, a notification arrives with a screenshot of the problem screen.
This way, an update in the target system is visible before it affects the production process, and the fix takes hours instead of weeks of downtime.
A backup processing route covers the case when the interface has been changed fundamentally: if recognition confidence is below a set threshold, the task goes through programmatic exchange, a file channel, or to an operator — data is neither lost nor duplicated.
We build area comparison and template matching on the open-source computer vision library OpenCV, so sensitivity thresholds are tuned to a specific screen rather than chosen at random.
The practical result: the layout lives its own life, while the scenario remains operational because it relies on the meaning of the element, not its coordinates.
Common concerns: resilience, access, and maintenance
The concerns among engineers and managers are usually the same, and they are valid: the robot works on top of the interface, so it depends on layout, access, and human behavior. We show how each risk is addressed at the level of the scenario and regulations, not promises.
How are false recognition positives filtered out?
The element is searched by a combination of features — its place in the form, a nearby caption, the field state — not by a single fragment of the image.
When confidence is low, the robot tries an alternative reference and takes a control screenshot; if there is no match, the step stops and goes to an operator rather than being performed at random. Disputed frames accumulate in the log and go into reference updates.
What access rights does the robot receive?
A separate account with the minimum required set of rights: only the sections and operations described in the scenario. This also removes the risk of duplicate data entry — a repeated run does not create a second request but reconciles with what has already been done. Every action is written to the log with a reference to the scenario step.
How are incidents investigated, and what about support after handover?
For each failure, a screenshot, step log, and scenario version are saved — they show in minutes what changed: an interface element, rights, or the response of an adjacent system.
We hand over a maintenance regulation: monitoring of runs, response to stops, and updating references after changes in the serviced system.
Let's discuss the task: we will return with an estimate and implementation plan
An accurate estimate of implementation timelines does not come from the phrase "we want a bot to repeat the operator's actions," but from data about the process and the environment where it will work.
Below is the minimum package that allows us to return with an automation work plan rather than a "from-to" range: the more specific the inputs, the fewer assumptions in the estimate and the higher the chance that the bot will go into production without reworking recognition.
- List of target systems — where exactly the process runs: a desktop client, terminal, legacy window without an API, web forms, scans, and email attachments. From this list we determine what is covered by computer vision and what by a standard programmatic interface.
- Access for discovery — a test bench or sandbox with an account for the required roles, a copy of reference data, and an anonymized sample of records. Without this, discovery turns into a survey rather than a measurement.
- Process description — the operator's steps in order, decision points where they manually reconcile data, and the rule by which they choose an option.
- Input data samples — real screens, scans, photos, and emails, including messy ones: overexposed, rotated, with handwritten notes.
- Volume and frequency — how many operations per shift, how the load is distributed during the day, and what window the bot must fit into.
- Nonstandard situations — what a person currently does when data does not match, and who in the company is responsible for changes in the target systems' interfaces.
- Acceptance criteria — how we agree to measure the result: the share of operations closed without manual intervention and the class of errors that is unacceptable for you.
If some data is closed, send what you have: we close the gaps during discovery and set them out as separate stages of the plan. Describe the task in free form and indicate by what deadline you need the result — we will return with an estimate and implementation plan.
