Development of AR object recognition and tracking (Object Tracking)

Object tracking: why the object track gets lost in an application

Object tracking breaks down not when the object leaves the frame, but earlier — when there are fewer reference features than needed for stable pose estimation.

A quick camera sweep produces motion blur and a drop in sharpness: features blur, and the tracker begins guessing the position rather than refining it.

A change of viewpoint adds a second constraint — on flat and monochrome areas there is nothing to latch onto, so the object's scale and tilt diverge from reality.

A separate class of failures is object occlusion: a hand, a passerby, or a neighboring object covers the reference area for a second or two, and the set of visible features drops to zero.

ARCore and related platforms enter relocalization at this moment, and until it completes, the anchor either freezes or jumps to a similar area nearby.

Low light makes all of the above worse: sensor noise and soft shadows create false boundaries, the object is found but with an offset of several centimeters.

The cost of these failures is measured not in frames but in user behavior. The virtual model slips off the exhibit, the hint lands away from the equipment node, the operator scans the shelf again, and some sessions end at the very first failure.

Support receives "the app doesn't work" — although the cause is track loss, while recognition and logic performed as expected.

In practice, this scenario is closed by a combination: robust recognition at the input, relocalization after occlusion, pose prediction between frames, and confidence thresholds at which the app honestly asks the user to point the camera instead of showing a shifted overlay. We'll break down how this is built and which capture scenarios should be tested before release.

What stable tracking provides: accuracy, speed, fewer failures

Stable object tracking is verified not by the feeling that "it seems to work," but by the share of successful tracks: on a set of test scenes, the object is followed to the end of the session in most runs, rather than lost midway.

The metric is calculated across the entire sample — different viewpoints, lighting, motion speed — not from one successful demonstration.

Time to first recognized frame also becomes predictable: the scene initializes in fractions of a second, and the user sees a hint before they have time to decide the app is unresponsive.

This is a result of model warm-up and preloading the object dictionary, not luck on a specific device.

The main effect is the failure classes that disappear entirely. The scene stops breaking under partial object occlusion, rapid camera movement, low light and glare, and false detections on similarly shaped objects are filtered out at the hypothesis selection stage.

Track stability is a separate metric: after a short occlusion, the object returns with the same identifier rather than being initialized anew. Without this, the object counter and the binding of hints to a specific item fall apart at the first occlusion.

Finally, operation on mid-range devices: the same share of successful tracks holds on mid-segment hardware, not only on flagship models. This is achieved by limiting the computational load per frame and adapting the processing rate to the current performance headroom.

Object tracking modes: comparison of approaches and limitations

The tracking mode is chosen along three axes: track stability, scenario freedom, and computational cost. Marker tracking provides predictable geometry — pose is restored from the corners of a marker of known size, and error does not accumulate over a session.

Feature-based scene tracking relies on natural points — corners, contours, textures — and does not require preparing the site.

The flip side: on monochrome and reflective surfaces there are not enough features, drift accumulates, and periodic revalidation against a reference frame is needed.

The hybrid scheme divides the load: pose refinement on the device, heavy revalidation in the cloud using sparse frames. During task analysis, we look at which mode will sustain your scenario over a long session.

How implementation proceeds: stages of engineering work

Implementing tracking is not connecting a library but tuning it to a specific scene: what works confidently on a test stand falls apart under different lighting or surface material. Therefore, we go from operating conditions to algorithm parameters, not the other way around.

1. Scene audit. We analyze how the object looks under real conditions: lighting changes, glare on glossy surfaces, background textures, camera movement speed, typical occlusions. The output is a list of conditions in which recognition must work.
2. Reference feature scheme. We determine what tracking latches onto: natural surface features, object geometry, or markers. We select features so that they are readable at an angle and at the working distance.
3. Tracking setup. We set trigger thresholds, filtering of anchor jitter, behavior under partial occlusion, and reinitialization rules after object loss.
4. Testing on real scenarios. We run the solution on site with the devices and lighting where it will live. We record the share of successful captures, recovery time after loss, and stability of anchored elements.
5. Refinement based on test results. We expand the feature set, change traversal viewpoints, add hints for uncertain capture — until the metrics reach the agreed level.
6. Solution handover. We deliver configuration documentation, the reference feature scheme, a set of test scenarios, and instructions for the team that will support the product.

Handover package: scene schemes, object sets, regulations

After acceptance, the customer's team retains not a demo prototype but a working system with a complete set of materials.

Everything we configured during the project is handed over in editable form — so that the next scene change or catalog addition does not require us as the sole holders of the knowledge.

• Recognition configurations — a profile for each site: trigger thresholds, allowed viewpoints and distances, behavior in low light. The profile is edited without rebuilding the app.
• Scene schemes — layout of capture zones, anchoring of reference elements, requirements for background and light. A new scene is deployed according to the scheme, not by trial and error.
• Object cards — reference samples, matching features, and boundaries beyond which the result is considered unreliable, with an explanation of the reason for each boundary.
• Object sets — collections for specific tasks, separated by versions: demo, working, archived, with revision history and an owner.
• Update regulation — the procedure for adding and removing an object: sample preparation, testing on a test scene, time to go live, conditions for rolling back to the previous set version.
• Incident review regulation — which events are recorded, how to distinguish a recognition failure from a scene error, who receives the request and within what timeframe.
• Solution documentation — architecture diagram, integration points with your systems, exchange formats, acceptance checklist, and a list of known limitations.

We agree on the package composition at the start of the project: you see in advance what will remain within your team and which documents it can use to work independently.

Implementation case: object tracking in a warehouse and retail

In a retail chain's distribution center, accounting relied on manual rounds: a storekeeper with a data collection terminal walked the zone and checked pallets against storage locations using a paper sheet.

One round took 25-30 minutes, a full warehouse reconciliation stretched over almost a shift, and discrepancies surfaced only during inventory.

In the sales floor the picture was the same — display was checked by eye, and an empty shelf was noticed a day or two later, when the customer had already left.

The pilot, which received the working name Sklad-Trek, was assembled as a camera app on the shift's standard smartphone.

The storekeeper points the camera at a pallet or shelf — recognition identifies the product and keeps the object in frame during movement, the counter recounts units and links them to the storage location. Data goes to the accounting system immediately, without manual entry or a second pass.

Metrics before and after

Tracking stability turned out to be the decisive detail: if a pallet was partially obscured or moved past a rack, the track did not get lost and continued to follow the object.

In low-light areas, we added local illumination and additional training on images from the warehouse itself — recognition reached 96-97% correct detections.

The shift stopped idling on search: downtime was reduced from hours to minutes, and the freed-up time went to receiving and handling defective goods rather than reconciliation. The same loop was transferred to retail — a floor round became one pass instead of two.

How tracking differs from recognition and are both needed?

Object recognition answers the question "what is this": the model finds an object in the frame and returns a class with a bounding box.

It remembers nothing about the next frame — on a repeat pass, the same object will receive a new number, and during a brief occlusion it will simply disappear and appear again.

Track persistence is identification between frames: the system predicts the object's position, matches it with the previous state, and maintains a single label while the object is in view. This yields a smooth bounding box without jitter and correct counting of unique objects.

The choice of mode is dictated by the task. If the answer "is there an object and which one" is needed, recognition is enough. If continuity over time matters — counting passersby, trajectory, interface reaction to movement — track persistence is indispensable.

In practice, we almost always build a combination of approaches, as is done in platform tooling at the level of ARKit: recognition periodically restarts the track on loss, and a lightweight tracker holds the object between restarts.

The share of frames where the heavy model runs drops, the image remains stable, and computational cost stops growing with session duration.

Technical questions: weak devices, light, scaling

We test low light and mid-range devices with one scenario: capture the same object as illumination drops and see at what point the track goes into search instead of a false box.

Relying on color alone does not work here — features are supplemented with contour geometry and motion, and the confidence threshold is configured so that a lost object is marked as lost rather than "sticking" to an unrelated item.

Growth in the number of objects and offline mode are solved architecturally. A limited number of active tracks are maintained on the device at the same time, the rest wait in a queue and are raised by zone priority; retraining for new classes does not increase the per-frame load.

The model and weights are stored locally, so recognition runs without a network, while the cloud is connected for analytics and model updates.

Will the app run on a mid-range device?

Yes. The model is quantized and runs on the neural accelerator, detection is launched not on every frame, and tracking between launches holds the position. We measure heating and battery consumption on the target device fleet before release, not after.

Does recognition work without a network?

Yes, the weights and tracking logic are stored in the app, so the object is tracked offline without a request delay. The network is needed for collecting statistics and additional training: accumulated data is sent when connectivity appears, while local results are not lost.

Let's discuss the task: we'll select a mode and come back with an estimate

To come back with a work plan and timeline estimate rather than general considerations, we need a minimum of input data.

The more specific the task statement, the more accurately we will select the tracking mode — by marker, by the object's natural features, or hybrid — and the fewer reworks there will be during integration with the app.

What to send for analysis:

• usage scenario: what enters the frame, from what distance and at what angles, whether the object moves or stands still;
• target objects and capture conditions — lighting, background, glare, occlusion by other items;
• requirements for track stability: holding under partial occlusion, camera shake, multiple objects at the same time;
• platform constraints: device fleet, operating system versions, whether offline operation is needed;
• what already exists — a prototype, source 3D models, design, access to the test stand;
• acceptance criteria in measurable form: share of correct recognitions, time to acquire track, allowable reaction latency.

Based on the discussion, you receive a work plan with stages and checkpoints, an architecture diagram with the load distribution between device and server, a timeline estimate for each stage, and a pilot launch scenario on a limited set of objects — usually enough to verify recognition quality before full assembly. Describe the task as it is, and we will come back with an estimate and plan.