GUI Automation with Computer Use AI Agent
Have you ever faced a situation where a legacy ERP on Windows has no API, and automation via RPA breaks after every update? Computer Use solves this: the agent sees the screen via screenshots and acts like a human. We develop such agents "turnkey" — from assessment to production with monitoring. Our team has 10+ years of experience in AI/ML and over 40 completed projects. We'll assess your task in 1 day — contact us.
Why Computer Use is More Reliable Than RPA
Classic RPA requires precise description of element paths (XPath, coordinates, CSS selectors) and a stable screen structure. Computer Use uses computer vision: the agent analyzes the screenshot and determines which actions will lead to the goal. This provides resilience to interface changes — if a button moves, the agent will still find it. Vision-based automation adapts to UI changes 5x faster than RPA.
| Characteristic | RPA | Computer Use |
|---|---|---|
| Adaptation to UI changes | Low (scripts break) | High (model sees changes) |
| Setup for a new task | Days-weeks | Hours-days |
| API requirement | Not required | Not required |
| Handling non-standard elements | Difficult (popup, drag-and-drop) | Possible with refinement |
How Computer Use Works
The basic principle: agent receives screenshot → analyzes interface → selects action → executes → receives new screenshot → repeats until task completion.
Example Python agent implementation (Claude + PyAutoGUI)
import anthropic import base64 import subprocess from PIL import ImageGrab import pyautogui import time import json from io import BytesIO client = anthropic.Anthropic() def capture_screenshot() -> str: """Takes a screenshot and returns base64""" screenshot = ImageGrab.grab() # Resize to save tokens screenshot = screenshot.resize( (screenshot.width // 2, screenshot.height // 2) ) buffer = BytesIO() screenshot.save(buffer, format="PNG", optimize=True) return base64.standard_b64encode(buffer.getvalue()).decode("utf-8") def execute_computer_action(action: dict) -> str: """Executes action on the computer""" action_type = action.get("type") if action_type == "screenshot": return "screenshot_taken" elif action_type == "left_click": x, y = action["coordinate"] pyautogui.click(x * 2, y * 2) return f"clicked at ({x}, {y})" elif action_type == "double_click": x, y = action["coordinate"] pyautogui.doubleClick(x * 2, y * 2) return f"double-clicked at ({x}, {y})" elif action_type == "right_click": x, y = action["coordinate"] pyautogui.rightClick(x * 2, y * 2) return f"right-clicked at ({x}, {y})" elif action_type == "type": pyautogui.write(action["text"], interval=0.05) return f"typed: {action['text'][:50]}..." elif action_type == "key": pyautogui.hotkey(*action["key"].split("+")) return f"pressed: {action['key']}" elif action_type == "scroll": x, y = action["coordinate"] direction = action.get("direction", "down") amount = action.get("amount", 3) if direction == "down": pyautogui.scroll(-amount, x=x * 2, y=y * 2) else: pyautogui.scroll(amount, x=x * 2, y=y * 2) return f"scrolled {direction}" elif action_type == "mouse_move": x, y = action["coordinate"] pyautogui.moveTo(x * 2, y * 2) return f"moved to ({x}, {y})" return f"unknown action: {action_type}" class ComputerUseAgent: """Agent for GUI automation via Computer Use""" COMPUTER_TOOLS = [{ "type": "computer_20241022", "name": "computer", "display_width_px": 960, "display_height_px": 540, "display_number": 1, }] def __init__(self, max_iterations: int = 50): self.max_iterations = max_iterations def run_task(self, task: str, context: str = "") -> dict: """Executes a task on the computer""" system = f"""You control the computer to perform the task. Use the computer tool to interact with the interface. Take a screenshot before each action to verify the current state. Report when the task is complete or if you encounter an insurmountable obstacle. {"Context: " + context if context else ""}""" messages = [{"role": "user", "content": task}] iterations = 0 actions_log = [] while iterations < self.max_iterations: screenshot_b64 = capture_screenshot() if messages[-1]["role"] == "user" and isinstance(messages[-1]["content"], str): messages[-1]["content"] = [ {"type": "text", "text": messages[-1]["content"]}, { "type": "image", "source": { "type": "base64", "media_type": "image/png", "data": screenshot_b64, }, } ] response = client.beta.messages.create( model="claude-opus-4-5", max_tokens=4096, system=system, tools=self.COMPUTER_TOOLS, messages=messages, betas=["computer-use-2024-10-22"], ) tool_results = [] task_completed = False for block in response.content: if hasattr(block, "text"): if any(kw in block.text.lower() for kw in ["task completed", "done", "готово", "completed", "done"]): task_completed = True elif block.type == "tool_use" and block.name == "computer": action = block.input result = execute_computer_action(action) actions_log.append({"action": action, "result": result}) time.sleep(0.5) new_screenshot = capture_screenshot() tool_results.append({ "type": "tool_result", "tool_use_id": block.id, "content": [{ "type": "image", "source": { "type": "base64", "media_type": "image/png", "data": new_screenshot, } }], }) if task_completed or response.stop_reason == "end_turn": return { "success": True, "iterations": iterations, "actions": actions_log, "final_message": next( (b.text for b in response.content if hasattr(b, "text")), "" ), } messages.append({"role": "assistant", "content": response.content}) if tool_results: messages.append({"role": "user", "content": tool_results}) else: break iterations += 1 return {"success": False, "iterations": iterations, "actions": actions_log} How to Implement Screen-Aware Automation in 1 Week
- Task Analysis — describe the scenario: which windows, which fields, expected result.
- Prototype — run a basic agent on your PC (3–5 days).
- Calibration — configure coordinates, retry logic, human-in-the-loop.
- Test — run 100+ sessions, collect metrics.
- Deploy — package into a service, add monitoring.
Practical Applications
Processing outdated desktop applications without API. An ERP system without API but with Windows GUI. The agent enters a form, fills fields, clicks "Post", gets the result — all via screenshots. Speed: 40–60 operations per hour vs 15–20 for an operator. Time savings reach 70%, reducing costs for maintaining legacy systems. Typical project cost starts at $5,000, with average annual savings of $30,000 in labor.
Data migration between systems. Export from old CRM → import to new one when there is no direct integration. The agent copies records via GUI of both systems.
Regression testing. The agent goes through user scenarios in a web application, checks results, logs anomalies.
Typical Problems and Their Solutions
- Model hallucinations. The agent may "think" a click was successful even though the UI hasn't changed. Solution: retry logic with state verification after each action.
- Sensitivity to screen scaling. If the screenshot is resized, coordinates may be inaccurate. We use automatic scale detection and calibration.
- Latency. One step takes 1–3 seconds. For long scenarios, this can be slow. Optimization: batch processing and reducing the number of screenshots.
Limitations and Real Metrics
Honest numbers from our practice:
| Task | Success Rate | Iterations |
|---|---|---|
| Fill a form with 10 fields | 87% | 8–15 |
| Navigate through 3+ screens | 71% | 15–25 |
| Work with popup/modal | 63% | 10–20 |
| Complex drag-and-drop | 41% | 20–40 |
For production, you need: retry logic, human-in-the-loop at low agent confidence, logging of all actions for audit.
Deliverables
- Documentation — architecture description, operation manual, parameter tuning guide.
- Training — 2–3 sessions for your team on how to fine-tune and maintain the agent.
- 3-month support — monitoring, bug fixes, adaptation to new application versions.
- Monitoring dashboard — success metrics, iteration count, p99 latency, anomaly logs.
We guarantee transparency: you see every step of the agent and can intervene at any moment.
Timeline
- Basic Computer Use agent: 3–5 days
- Specific automation task (one form/workflow): 1 week
- System with retry and human-in-the-loop: 2 weeks
- Production deployment with monitoring: 3–4 weeks
Assess your task — contact us, we'll show a demo on your application. Get an engineer's consultation: fill out the form on the website with a description of your GUI scenario, and we'll assess automation feasibility in 1 day.
Anthropic Computer Use beta documentation