AR App Development: Where a Project Breaks at the Start
AR app development rarely breaks in a demo and almost always breaks at a real site. In a demonstration, there is a flat table, good lighting, and slow camera movement, so everything holds up.
In the field, the scene behaves differently: scene tracking loss happens on a solid-color wall, during a sharp turn of the device, or in a room with glare and mirrored surfaces.
The camera stops recognizing the environment, the coordinate system is rebuilt from scratch — and the object that was just standing on the table drifts away from the surface by centimeters, and sometimes by meters.
The second class of failures is drift of 3D objects. Anchoring to an anchor without regular verification against recognized geometry accumulates error: the object slowly "floats" along the plane, changes scale and tilt.
In a short session this is unnoticeable; in a long one, the discrepancy becomes obvious, and no careful 3D model compensates for it. The cause is almost always in the scene architecture, not the content: the anchor is placed once and never rechecked.
The third is device overheating. The camera, rendering, and recognition work simultaneously and continuously; the thermal budget of an average device runs out faster than it seems.
The system first reduces the frame rate, the image begins to lag behind hand movement, and then the app closes. On flagship devices the problem is masked, so it is easy to miss during testing and then encounter it across a fleet of ordinary phones.
All three failures lie in the same plane — in how the session loop is structured, how anchors are chosen, and the per-frame load budget. Below we break down how this is addressed at the level of architecture and device profile, rather than with cosmetic fixes.
What Changes After Implementation: Five Measurable Effects
The first effect is keeping the virtual object in frame. Previously, the model slid off the anchor point when the camera turned: the anchor was lost, and the object had to be placed again.
Now the position is recalculated from the scene geometry, the object stays in place while the user walks around it, and the error accumulated over a session is measured in centimeters and does not grow over time.
The second is tracking stability in real conditions: low light, solid-color surfaces, sharp turns of the device.
The margin for these scenarios is built into the prototype, so the report shows not an average number but the share of sessions where tracking was never lost and the object did not "slide off".
The third and fourth are cold-start time of the AR mode and response to user action. The first transition to the camera takes a couple of seconds because models and textures load in the background and do not block the scene.
Response to touch is fractions of a second: for try-on or aiming at an object, delay reads as "broken," not as "slow."
The fifth is the share of devices where the feature works predictably: this is a device fleet that has passed testing on a live build, not a list from documentation.
This also addresses an entire class of failures — desynchronization between camera position and virtual scene, which makes the app show the object off target or go to a black screen.
Implementation Options: Native Module, Engine, or Web
The three approaches differ not in image quality but in where augmented reality is embedded and what price you pay in maintenance.
A native AR module relies on platform SDKs — ARKit and ARCore: access to plane tracking, lighting estimation, and the camera is complete, but the logic is duplicated for each platform.
A web scene in the browser removes the installation barrier: the user follows a link and immediately sees the model in frame. The price is limited access to sensors, performance drops on heavy scenes, and dependence on the browser version. Below is what fits which scenario.
| Option | When it fits | Limitations |
|---|---|---|
| Native AR module | AR complements an already released app | Two codebases, separate releases per platform |
| Cross-platform AR app | Unified code and a complex 3D scene | Noticeable build size, lag behind new platform capabilities |
| Web scene in browser | Fast launch without installation, promo and demo | No access to some sensors, weaker on heavy scenes, browser dependence |
| Hybrid implementation | Both reach and tracking quality are needed | More complex build and testing: two rendering branches |
The hybrid scheme is chosen when the web scene covers fast entry and the native module handles heavy scenes inside the app.
This option is more expensive to maintain, so we fix the boundary between the branches before development starts — otherwise behavioral divergence surfaces among users.
Work Stages: From Task Definition to Handover
We structure AR implementation stages so that technical risks are removed before the bulk of the work goes into the project: the most expensive error is one in the scene schema discovered at the final build. That is why the first steps are about the scenario and geometry, not code.
- Scenario analysis. We fix where the camera points, what the user does with their hands, and what action counts as success. The output is a measurable acceptance criterion, not "it should look impressive."
- Scene schema. We lay out anchors, reference planes, interaction zones, and object scale relative to human height. The scene schema becomes a shared document for design, development, and acceptance.
- AR feature prototype. We build a minimal version on one device model: we check that the scenario is reproduced in a real room, not only on test markers.
- Tracking and occlusion. We configure surface recognition, occlusion of virtual objects by real ones, and scene behavior when tracking is lost — this determines whether the image "holds up" during movement and in low light.
- Testing on a device fleet. We run the build on different camera generations and performance levels (ARKit and ARCore behave differently in the same room). We record where stability drops.
- Acceptance. We go through a checklist by scenario and device, compare against the criteria from the first step, and prepare a protocol with known limitations.
- Handover. We hand over source code, the scene schema, the update policy, and instructions for adding new models — the client's team can develop the AR block without us.
What We Hand Over: Build, Schemas, Policies
Handover is not an archive with source code but a set of artifacts by which your team or our support engineers reproduce the build, edit the scene, and update content without deciphering someone else's code. Below is what is included in the AR module package.
- Working build — assembled app variants for target platforms, environment configuration, and scripts that reproduce the build from scratch on a clean machine.
- Scene schema — node hierarchy, anchor and placement point binding, plane recognition logic, and object behavior when tracking is lost.
- Content access rules — roles and permissions: who publishes and unpublishes models, where source files are stored, and how versioning and publication rollback work.
- AR module documentation — extension points, contracts between layers, platform component dependencies, and their update procedure.
- Model update policy — requirements for formats and optimization, pre-publication checks, and the procedure for replacing content without rebuilding the app.
- Support guide — typical failures, what to look for in logs, how to roll back the app and restore the previous scene version.
- Access matrix — where signing keys and publication accounts are stored, who owns them, and how they are transferred when changing contractors.
The package is updated with every change to the module: if a change is not reflected in the scene schema or policy, the handover is considered incomplete.
Case Study: Virtual Try-On in an App — Before and After
In the app of the clothing chain Модный дом, try-on was a separate screen reached by one shopper out of seven.
The scene with garments took 4–5 seconds to open, the model "floated" relative to the body, and it cut out when the camera turned; in the sales floor in low light, tracking was lost almost immediately.
According to logs: 62% of try-on sessions ended in an error, average duration was 8 seconds, and there were almost no add-to-cart actions from the try-on.
What We Rebuilt in the Scene and Tracking
The garments were rebuilt for mobile output: simplified geometry and pre-prepared textures instead of heavy source models — the scene weight dropped several times over, and the first frame appears noticeably sooner.
In tracking, we added body position prediction and smoothing — the garment stays in place during fast movements and turns instead of "peeling off." Instead of one universal model, we keep several fits and select them based on the shopper's measurements, so try-on is no longer "by eye."
The lighting threshold was set based on measurements in real sales floors: instead of an error message, the app suggests where to stand. This removed the most common failure scenario — the shopper did not abandon the try-on but completed it.
Metrics Before and After
The share of successful try-on sessions — from 38% to 94%. Time to first frame — from 4–5 seconds to 1.5 seconds.
Average try-on duration grew from 8 seconds to a minute and a half, the share of add-to-cart actions after try-on tripled, and returns with the reason "did not fit" decreased by about a quarter.
How AR Try-On Differs from a 3D Preview on a Website
A 3D preview lives inside the screen: the model rotates on a neutral background, and the user imagines the scale themselves — whether the item will fit next to the sofa, whether it will match the shelf in height.
AR try-on removes this assumption: the camera shows the room, the app recognizes the plane, and places the object there in real proportions.
Technically, the difference is in surface anchoring. The app determines the plane, tracks the device position in space, adjusts scale, and accounts for lighting, so the object does not float or sink, and the shadow falls under the legs. This answers the question "will it fit," which a rotatable model cannot answer.
The use scenario determines the choice, not the technology itself. Shoes, watches, accessories, packaged electronics — shape and color are enough with a rotatable model: it loads quickly and opens on any device.
Furniture, large appliances, decor, and anything tried on "on yourself" require surface anchoring: there, dimensions are decisive.
User behavior differs.
Preview increases depth of viewing and the number of comparisons; AR try-on increases the share reaching checkout and reduces returns due to "did not fit the size." The price is extra steps: camera permission, aiming at the surface, waiting for recognition, and some of the audience is filtered out by them.
That is why we do not pose an "either/or" choice.
In a typical scenario, preview covers entry into the catalog, and AR try-on covers the decision stage for items where dimensions matter; the split is set by category, and the rest of the app architecture is calculated from it.
Technical Concerns: Devices, Light, Accuracy
Scene limitations are set not by code but by the combination of camera, inertial sensors, and platform libraries. Below are four questions that most often come up during acceptance, and how we address each.
On which models will the app even launch AR mode?
The scene relies on the platform layer — ARKit and ARCore of current branches, so we fix the device list before development starts. We maintain two tiers: advanced on models with a depth sensor, and basic on the rest, where tracking runs on the camera and gyroscope.
For models outside the list, the app installs but shows a hint instead of the scene — this is more honest than a failure at launch.
What happens in low light?
The camera does not find feature points: the scene "floats," objects jitter and shift. We display a tracking quality indicator, ask for more light, and switch the object to plane-anchoring mode instead of frame-by-frame alignment. Below the lighting threshold, we show a clear message rather than artifacts.
How accurate are measurements in the scene?
The error depends on distance and surface texture: up close on a marked plane it is a few centimeters; on distant and solid-color areas it grows.
Where accuracy is critical, we limit the working area and introduce a reference marker of known size — this way the error does not accumulate as the user walks around.
Does the scene work without a network?
Recognition and tracking are computed on the device, so offline the scene launches and holds. The network is needed for content and analytics: we package materials into a local cache, accumulate events, and send them in a batch when connectivity appears.
Heavy content loads in the background in a Wi-Fi zone so that in the field the user does not wait for loading.
Let's Discuss the Task and Return with an Estimate and Work Plan
For the AR project estimate to be substantive rather than "by eye," we need inputs on the task: what we recognize, under what conditions it should work, and on which devices.
Augmented reality is sensitive to the scenario — marker-based, plane-based, and coordinate-based tracking have different labor intensity and different requirements for the content that will need to be prepared.
What to send to start:
- scenario description: what the user sees in the camera and what happens after recognition;
- list of devices and minimum mobile OS versions to support;
- materials — models, photos, videos, drawings; if you do not have them, say so, and we will discuss content sources separately;
- 2-3 reference apps and an explanation of what exactly you like about their behavior;
- constraints: event or launch date, data access requirements, existing infrastructure;
- whether there is an existing app into which AR is embedded, or the build is from scratch.
We will separately clarify what usually comes up during the work: how the scene behaves in low light, whether offline operation is needed, what happens when tracking is lost, and how the app returns the user to the scene.
In response, we will return with a work plan: team composition with roles, stages and their duration, what we do at each stage, and when you accept the result. As a separate item — scenario risks and a buffer for them. Describe the task in free form — we will sort it out further during the dialogue.