Why a desktop AI agent breaks after an interface update
An agent recorded via click coordinates and a path in the window tree lasts only until the next update of the business application.
Developers move a button to another panel, change the field order, add an intermediate confirmation dialog — and the scenario that had been closing entries for a month stops finding elements.
The same effect comes from a change in resolution or scaling on the workstation: the coordinate grid shifts, and the click lands in an empty spot.
The worst part is that such a failure is silent. The agent does not crash with an error or write an alarming line: it waits for an element, repeats the step, postpones it, and moves on to the next one, while some documents remain unprocessed. Externally, the process looks alive; in reports, there are zero errors.
The failure is detected not in the agent logs but in the accounting system: discrepancies in the register, stuck requests, unposted items that surface during reconciliation. By this point, a batch has accumulated, and it has to be handled manually.
The fragility of an automation scenario is not a property of a particular application but a consequence of how it is built.
An agent that relies on element semantics via UI Automation, verifies the result of each step, and reports a discrepancy before it reaches the register can survive an application interface update without being rewritten from scratch. Let's break down how this is addressed.
What agentic automation delivers: the effect in measurable numbers
The effect of agentic automation is measured not by the number of launched scenarios but by the time that stops being spent on the operational flow itself.
The agent takes over transferring data between windows, exports, and filling out forms — the operator stops being a link in information transfer and focuses only on decisions that require a human.
The speed of request processing increases not by speeding up a single step but by eliminating the queue: while an operator handles one request, the agent concurrently processes several dozen similar ones and does not wait for confirmation on each.
This is measured by the time from request arrival to the finished result and the share of requests closed without human involvement.
The share of manual input errors falls for an obvious reason: the value is transferred from source to destination directly, without keyboard entry and without visually comparing two screens. Errors stop depending on end-of-shift fatigue.
Repeated scenario runs also become a controllable quantity: each step gets a completion flag, and after a failure the agent continues from a checkpoint rather than going through the entire path again.
This matters where a repeated run creates duplicates — in accounting systems, when sending documents, when creating tasks.
A class of failures that disappears entirely is counted separately: an empty field, an incorrect date format, a lost row in an export. They are not mitigated but disappear because format validation happens before writing.
Failures of external systems remain — their handling is designed as separate branches of the scenario.
Agent operating modes: comparison of approaches and limitations
Agent compatibility is determined not by the model but by where it runs. Agent launch modes differ in account permissions, session activity, screen resolution, and version update method — and this is most often what determines whether a scenario starts or fails midway.
An agent on the operator's workstation runs in a live user session: it sees the same windows, the same input focus, and the same environment as a human. The limitation is dependence on session state: steps fail when the screen is locked or a window is overlapped.
A dedicated machine for the agent removes this with a persistent session and predictable environment, but requires separate access to the target system and monitoring of the node's state.
| Option | When it fits | Limitations |
|---|---|---|
| On the operator's workstation | Piloting a process, tasks with human confirmation | Depends on session activity and screen settings, conflicts with the operator's parallel work |
| Dedicated machine for the agent | Regular flow of similar tasks without human involvement | Requires separate credentials, monitoring of node availability and agent updates |
| Shared fleet with centralized management | Dozens of agents and scenarios for one system | Requires a task registry, separation of permissions between scenarios, and a queue for shared windows |
In practice, modes are combined: a centrally managed fleet maintains uniform agent versions and a shared step log, while sensitive operations remain with the operator. Describe your scenarios — we will come back with an assessment of which option will support them without rework.
Implementation stages: audit, design, setup, validation, handover
Implementation begins not with a showcase launch but with an analysis of how the task is performed today.
We document the sequence of steps, their frequency, input variability, and failure points: where a human makes a decision, what happens with a non-standard system response, which windows appear less than once a week.
Automation usually breaks precisely on these rare branches — that is why they are described during the audit rather than surfacing in pilot operation.
- Process audit. We break down the current scenario step by step, compile a list of variations, and separately document exceptions and rules for access to accounts and reference data.
- Scenario design. We agree on a map of the agent's actions: what it does itself, where it stops and hands control to the operator, and how it responds to interface changes in target applications. The design is agreed in writing — it becomes the basis for acceptance.
- Setup and supporting infrastructure. We assemble the environment in an isolated contour, configure access on the principle of least privilege, logging of every action, and storage of screenshots for incident review.
- Validation on real examples. We run the scenario on accumulated transaction history and on artificially created failures: an unavailable resource, a changed field, a response timeout. We separately check how the agent terminates on interruption — without "hung" windows or unclosed operations.
- Pilot operation. We launch it paired with an operator on a limited area: the agent performs the work, and the human verifies the result. Discrepancies are analyzed, and rules are refined until the share of manual interventions is acceptable.
- Handover of support regulations. We provide the scenario description, instructions for restarting and typical failures, and the update procedure for changes in target applications.
After handover, the agent remains manageable: changes go through the same validation contour, so an update does not turn into a one-off emergency.
What the customer retains: configurations, policies, regulations
After delivery, the customer retains not only a working agent but also a set of artifacts that let the team change scenarios, grant access, and troubleshoot failures on its own without contacting us. The set is designed for your team to handle support, with us joining on a targeted basis.
- Agent configuration — scenario steps, window element search methods, wait times, and retry rules are moved into separate files: editing a step does not require rebuilding code.
- Process design — how the agent completes the entire scenario, where the branches are, and what data passes between steps, so a new engineer can understand the logic in one pass.
- Access policies — which account the agent runs under, which systems and directories it accesses, and what is explicitly prohibited to it.
- Run log — for each launch: step number, result, duration, screenshot at the moment of error; an incident is analyzed from the record, not from memory.
- Set of validation runs — scenarios on the test bench that are run after every configuration change and catch regressions before the production contour.
- Operator instructions — how to launch, pause, resume from a failed step, and what to do with a non-standard window.
- Update regulation — who edits the configuration, who approves it, and how to roll back to the previous set of steps on regression.
Case study: an agent instead of manual request entry into the accounting system
Before
Requests arrived by email and messaging apps, and were entered manually into the accounting system on 1C: the operator opened a form, transferred the details, checked the counterparty against the reference data, and selected the item.
One request took 7–9 minutes; during peak hours the queue stretched to the end of the shift, and discrepancies in details were found already during approval — a return and another round.
After
The agent parses the incoming message, matches the counterparty and items against reference data, and works in the accounting system windows like a human: clicks, field entry, switching between forms — without changes to the configuration or direct access to the database. It does not make assumptions on disputed cases but places them in a queue for operator review.
Metrics were captured during pilot operation on a live flow.
The share of manual rework after the agent was 9% versus 42% before it; average request time dropped from eight minutes to forty seconds, and detail discrepancies stopped reaching approval: the agent validates required fields before writing. Operators switched to exceptions and reference-data conflicts.
Every agent action is logged and reproducible, the screen is available to the operator, and control can be intercepted at any moment — that is why pilot operation was conducted on the live flow, without duplicate manual entry.
How a desktop AI agent differs from macro recording
Macro recording captures a sequence of actions: click coordinates, pauses, window position.
Such a scenario is reproduced literally — the panel shifted, the field order changed, a notification popped up, and the chain falls apart midway, when part of the data has already been sent to the system.
A desktop AI agent works not with coordinates but with the meaning of the screen. It analyzes the interface state — which fields are filled, what remains required, which window is on top — and decides which action leads to the goal.
Form state recognition is based on content and element structure, so moving a button elsewhere breaks nothing.
Rigid scripts are reliable only until the first interface change or non-standard state. Maintaining them is expensive: every accounting system update turns into editing branches, and an unforeseen modal window stops the entire process until manual intervention.
A separate difference is recovery after failure. A macro on an unexpected dialog simply waits or crashes with an error.
The agent notices a discrepancy with the expected state, repeats the step, closes the extra window, and returns to the checkpoint, while the entire action path remains in the log for analysis.
This does not mean the agent replaces everything. Where the sequence is strictly defined and the screen is stable, deterministic code is more predictable. We usually build a hybrid: the agent handles recognition and branching, while a proven script performs repeatable operations.
Technical concerns: resilience, access, rollbacks
The agent works on top of application interfaces rather than through their internal integration points. Hence the main question: what happens when the wrong window opens, a button moves, or the session drops midway.
The cycle is built with state verification after each step — the agent confirms the result by fact, not by click.
If the expected state does not occur, the step is repeated a limited number of times, after which the task is marked as requiring a human: the queue is not clogged with silent errors.
We grant minimal permissions: a separate account, access only to the necessary systems and folders, without administrator privileges on the workstation.
Every action is logged — what was opened, what was entered, where it stopped — and the course of any task can be reconstructed from it.
Implementation is reversible: the agent does not displace previous automation but stands alongside it; the share of operations is switched by routing and can be returned back in the same way. Disabling means stopping the scenario and revoking access; the data remains in place.
What happens if the application interface is updated?
The agent relies on several element attributes rather than coordinates. When something moves, it stops on a mismatch and reports which step broke — we make a targeted fix without rewriting the scenario.
What permissions does the agent need on the workstation?
Exactly those required for the target operations, and not one extra: a regular account, a separate profile, without access to payment and HR systems. We agree on the scope of permissions before launch.
Can we return to the manual process?
Yes. The previous order is preserved during implementation, and switching happens in parts. If the pilot group shows worse results than before, we return the route to humans without redoing what has already been done.
Let's discuss the task: where we start and what we need from you
For the assessment to be substantive rather than a rough guess, we need access to how the process works now. Usually, a scenario description, a test contour with the same system versions, and a list of applications in which the agent operates are enough. We gather this in one pass — and come back with a stage-by-stage implementation plan.
- Process description: operator steps, decision points, what is considered successful completion, and which errors are critical.
- List of systems: where the work is performed, how login is done, whether programmatic interfaces exist or only screen input.
- Test contour: a copy of the environment with anonymized data, accounts with the required permissions, access over a secure channel.
- Examples of complex cases: rare scenarios, pop-up windows, interface changes, manual confirmations — they show where the agent will hit a wall.
- Regulations and limitations: which actions require human confirmation, what is prohibited to the agent, who is responsible for the final decision.
- A responsible person on your side: someone who confirms scenarios and accepts the result of each stage.
- Target metrics: time per operation, share of scenarios without manual intervention, acceptable error rate.
Then it is our turn: we review the materials, reproduce the scenario on the test contour, and come back with a work estimate and implementation plan.
In the plan, we document what we deliver after each stage and how we roll back if the agent's behavior deviates from what was agreed.
