AI Object Tracking for VFX
We develop AI-based object tracking systems for VFX that solve the problem of track loss under occlusion and motion blur. Classic tools like Mocha Pro and PFTrack require manual point placement and break when blur exceeds 15 pixels. Our neural network solutions (CoTracker, DiffusionTrack, FoundPose) provide stable tracking even with 70% occlusion. We have 5+ years of experience in VFX tracking and over 10 successful projects for commercials and films. Neural tracking is 3x faster than classic methods with similar accuracy, and pricing is calculated individually based on material complexity. Post-production budget savings reach up to 60%.
Tracking Types and Tools
2D point tracking — tracking reference points for stabilization or match move. Modern neural tracking with CoTracker (Meta) tracks 256+ points simultaneously, accounting for mutual dependencies. It uses a transformer with temporal attention, providing robustness under partial occlusion. — Meta Research
Planar tracking — tracking a plane (wall, car side, screen) for graphic insertion. Neural version: DiffusionTrack, or a hybrid of homography + deep features from SuperGlue for matching.
6DoF object tracking — tracking 3D position and orientation of an object. FoundPose, FoundTrack work with a CAD model or learned prior.
Human body tracking — MediaPipe Holistic (33 skeleton points + hands + face), OpenPose, SMPL-X for full 3D body reconstruction from monocular video. Used for markerless motion capture.
Why Neural Tracking Is More Robust Than Classic?
Classic feature-based tracking (Lucas-Kanade, KLT) loses points at motion blur >15 px, occlusion >40% of the object, or sudden lighting changes. Neural approaches win through learned descriptors and visibility prediction. For example, CoTracker maintains tracking up to 70% occlusion — the model predicts whether a point is visible in the current frame and doesn't attempt to search where it isn't. When the object re-emerges from occlusion, tracking resumes via re-identification.
| Parameter | Classic (Lucas-Kanade) | Neural (CoTracker) |
|---|---|---|
| Motion blur >15 px | Point loss | Stable up to 30 px |
| Occlusion >40% | Track failure | Maintains up to 70% |
| Sudden lighting change | Drift | Stable |
| Number of points | 10–50 | 256+ |
import torch
from cotracker.predictor import CoTrackerPredictor
model = CoTrackerPredictor(checkpoint='cotracker2.pth')
model = model.cuda()
# video: (1, T, 3, H, W) tensor
# queries: (1, N, 3) - (t, x, y) for each point
tracks, visibility = model(video, queries=queries)
# tracks: (1, T, N, 2) - coordinates of N points in all T frames
# visibility: (1, T, N) - visibility probability
How to Integrate Tracking into a Post-Production Pipeline?
We output tracking in formats compatible with industry software: .nuke scripts or .abc (Alembic) files with animation curves for Nuke, After Effects, Blender. For match move, we use .chan format with camera transform data. Automation: Python API for Nuke allows direct writing of tracking data into Tracker4 nodes without manual transfer. This speeds up integration and reduces error risks when handing off between departments.
Step-by-step:
- Prepare the video file and define target objects.
- Choose tracking type (2D, planar, 6DoF) based on the task.
- Run the neural model on GPU — average processing time for 5 minutes of video is ~2 hours.
- Verify track quality via visualization.
- Export data in required format.
- Integrate into the scene via plugin or script.
Details about CoTracker
CoTracker is a transformer-based model from Meta. It processes all points simultaneously using attention over time and space, providing robustness to occlusions and blur. Weights can be downloaded from [GitHub](https://github.com/facebookresearch/co-tracker).What's Included
- Material analysis and approach selection (2D/planar/6DoF/human)
- Neural model development and tuning for the specific scene
- Tracking execution with intermediate iterations
- Data export in required formats (Nuke, AE, Blender)
- Documentation of used models and configuration
- Training your team on the delivered scripts
- Technical support during post-production
Timelines
| Task | Volume | Time |
|---|---|---|
| 2D point tracking | 1–5 min video | 1–3 days |
| Planar tracking with mesh | 1–5 min | 2–5 days |
| 6DoF object tracking | 1–3 min | 3–7 days |
| Full match move + camera solve | Scene 1–10 min | 5–14 days |
Pricing is calculated individually based on material complexity. We guarantee track deviation under 1 px on verified frames. Contact us for a project evaluation — we'll select the optimal turnkey solution. Request a consultation to discuss details.
From Our Practice
A 45-second commercial with a camera moving alongside a car on a highway. A logo needed to be "stuck" to the door, accounting for reflections and body deformation due to vibration. Challenge: speed 120 km/h, motion blur at 1/60s shutter, periodic sun flares. Solution: planar tracking via homography with SuperGlue-matching (robust to blur thanks to learned descriptors) + mesh-based deformation (door divided into an 8×4 grid, each node tracked independently) + lighting estimation via EfficientLit. Result: stable tracking over 1080 frames with deviation <0.8 px, 6 hours of automated processing instead of 3 days of manual tracking in Mocha.