Training ArcFace: why the model confuses faces on live footage
A model on a holdout set shows accuracy above 99%, but at the access control point it starts confusing people. The gap arises because the metric is calculated on prepared frames: even lighting, face in full frontal, person looking at the camera.
The live stream is a different distribution, and ArcFace and FaceNet are equally sensitive to this discrepancy if training and fine-tuning were not done on the right data.
There are three sources of errors. The first is similar faces: twins, relatives, employees in identical gear, helmets, or masks, when only a third of the face remains visible.
The second is frame quality: backlight from the entrance, motion, motion blur from shutter speed, a three-quarter angle instead of full frontal.
The third is outdated labeling: some reference images were taken a year ago, the person has changed their hairstyle, grown a beard, or started wearing glasses, but in the gallery they are still the same as before.
At the access control point, this results in two classes of failures. False triggers toward "do not let in" — an employee stands at the turnstile, a queue builds up, security manually confirms passage.
False triggers toward "let in" — an unauthorized person passes by coincidence, and this is already a security incident, not an inconvenience.
Face recognition errors are not cured by a single threshold: lowering the threshold reduces denials but opens up access.
We will examine how this is addressed — by tuning the threshold for the task, fine-tuning on real frames from the access control point, and controlling pairs of similar employees in the gallery.
Measurable effect: accuracy, error rates, verification time
It makes sense to compare models on frames from the site itself, not on public datasets: different optics, installation height, lighting, and typical angles produce a different picture.
Therefore, before adaptation, we run the baseline model over the archive and record the initial metrics — we then compare the result against them.
After fine-tuning ArcFace on labeled frames from the site, recognition accuracy on the validation set increases, and the false pass rate drops severalfold.
The model begins to work confidently in evening light, at partial rotation, and with face occlusion, where manual passage used to be triggered.
The key parameter is the threshold. By shifting it, we change the balance between false passes and false triggers: for access control at the checkpoint, it is more important not to let an outsider in; for time tracking, it is more important not to lose one of your own. We select the threshold from the curve on the holdout set and fix it as part of the configuration.
We measure verification speed on the stream: how many people pass per minute, what is the decision latency per frame, and how the system behaves with several cameras simultaneously.
In practice, this is fractions of a second per person — the queue at the turnstile stops building up.
We test robustness not only on our own archive: we run the model on shifted data — a different camera, a different time of day, seasonal clothing. If the metrics drop, we add such frames to the labeling and repeat the cycle.
We record the metrics in a report so that growth or degradation after each iteration can be seen in numbers, not in words.
Implementation modes: custom model, ready-made, hybrid scheme
The three approaches differ in how much data is needed before starting and how deeply one can influence the model's behavior.
A ready-made open model based on ArcFace or FaceNet gives the first working result on a typical stream; fine-tuning adapts it to specific footage, and training from scratch is chosen when neither of the first two options works.
The choice of approach is a trade-off between launch time, amount of data labeling, and control over the model.
| Option | When suitable | Limitations |
|---|---|---|
| Ready-made model | Full frontal, predictable lighting, one or two angles, quick launch needed | Accuracy drops on headwear, backlight, strong tilts; we do not change the internal logic |
| Fine-tuning the model | There is an in-house corpus of images from the target stream, shooting conditions are stable | Need labeling of "own/foreign" pairs and a holdout set, otherwise the model overfits to a narrow set |
| Training from scratch | Atypical capture channel or strict requirements for control over the architecture | Need a large and diverse corpus, timelines measured in months, higher risk of errors during data collection |
We make the decision based on measurements, not general considerations: first we run the ready-made model on the customer's real frames and calculate the false pass rate and false trigger rate at the selected threshold.
If the metrics fall short of the requirements, we move to fine-tuning — it closes the gap on a sample an order of magnitude smaller than training from scratch. We leave training from scratch for cases where the architecture itself is tailored to a non-standard capture channel.
Training ArcFace for your scenario: stages of work
Training ArcFace is not about running a ready-made library, but about fitting the metric to your scenario: angles, lighting, acceptable false match rate.
We start with an audit of frames and finish with the handover of the model with fixed thresholds, so that the result can be reproduced on your side. Below are the training stages.
- Data audit — volume and quality of frames: face size, angles, occlusions, defect rate. We count the number of people and frames per person, add rare cases, and remove duplicates.
- Labeling and preparation — alignment by key points, uniform scale, fixed split into training, validation, and holdout sets. The holdout set does not participate in training.
- Architecture selection — we compare ArcFace and FaceNet on your data: where similar people are separated better at the same response time. For ArcFace, we select the angular margin value.
- Training and monitoring — we add distortions in brightness, rotation, and blur, and monitor class separability on validation. We stop training when quality stops improving.
- Threshold tuning — using the error curve, we select the threshold for the scenario: checkpoint, control zone, or operator confirmation.
- Model validation on the holdout set — we test on people, sessions, and cameras the model has not seen, and look at metrics separately by subgroups.
- Deployment of recognition — we hand over the weights, configuration, inference module, threshold and monitoring regulations, and help connect it to your infrastructure.
Artifacts: model weights, labeling, regulations
Along with the trained model, the customer receives not a "black box" but a kit that allows operating the solution independently or handing it over to another team without losing context.
We consider the handover complete only when a third-party engineer can reproduce the result and make changes using these materials.
- Model weights in an open format — final weights after the last epoch plus checkpoints of several best epochs: they provide a margin for rollback and fine-tuning without re-running the entire training.
- Data preparation and training scripts with fixed parameters: composition and order of augmentations, learning rate, layer freezing strategy, initialization values — so that the result is reproduced to the last digit.
- Labeling in its original form: images, labels, split into training, validation, and holdout sets, log of excluded frames, and labeling quality control protocol.
- Metrics report: quality on the holdout set, operating point by threshold, false pass rate and false trigger rate, breakdown of errors by shooting conditions — low light, angle, occluded face.
- Operation documentation: launching recognition, threshold tuning for the desired error balance, frame requirements, procedure for processing embeddings and storing biometric data.
- Retraining regulations: which events trigger rebuilding — increase in denial rate for new people, change of cameras or shooting conditions, expansion of the employee list — how often scheduled checks are performed and how quality degradation is recorded.
- Rights to the model: legally documented transfer of rights to the weights, derivative models, and labeling, as well as a list of licenses for open components used in the solution.
Retraining regulations are not a formality: without them, the model quietly becomes outdated as the data stream changes, and the first signal comes from the security service.
Rights and documentation cover the second risk — when the team that trained the model is unavailable, and there is no one to understand the artifacts.
Case study: checkpoint with two scenarios, metrics before and after
At a production site, there are four checkpoints and about two thousand employees.
Access control worked on a model trained on an open face dataset: for similar employees (relatives, identical workwear, glasses, helmets), it produced false denials, and security let people through manually.
During peak hours, a queue built up at the turnstiles. Over a shift, more than a hundred manual checks accumulated, and some employees eventually started swiping their pass without looking at the camera — the system statistics stopped reflecting reality.
What was changed
We collected a dataset on the cameras of the checkpoints themselves — angles, lighting, seasonal clothing — and fine-tuned the ArcFace embeddings on our own labeling.
We left FaceNet as a second verification branch: a decision passes when both models agree, so the class of errors on similar faces is closed.
Thresholds were selected separately for each point — at the gate with side lighting, frame quality is different from that at the entrance turnstile. Comparison is done over a series of frames, not a single image, so glare or a head turn no longer turns into a denial.
Metrics before and after
| Metric | Before | After |
|---|---|---|
| False denial rate | 7% | 1.2% |
| Manual checks per shift at peak | more than 100 | about 15 |
| Passages without stopping | 82% | 97% |
The labeling and selected thresholds are reused at the remaining access points: each subsequent checkpoint is connected faster than the previous one and without collecting a dataset from scratch.
What to do if another model is already running at the site?
A running system is not just model weights: around it, face templates, selected decision thresholds, integrations with the accounting system, and rules by which an event turns into access have already been assembled.
Therefore, the fork of "replace or extend" is decided not by the quality of the model as such, but by what exactly stopped coping in the current setup.
If the degradation is local — new angles, lighting, an added camera — fine-tuning ArcFace on this data solves the task without rebuilding the template database.
Extending preserves the feature vector, so previously registered templates remain valid, thresholds hardly shift, and integrations with the accounting system are not touched.
Replacing the model entirely is a different scenario: the new feature vector is incompatible with the old one, so templates are recalculated from scratch, thresholds are calibrated on a labeled set, and decisions are compared with the previous setup until discrepancies fall within an acceptable range.
This path is justified when the model loses on a class of tasks, not on individual shooting conditions.
The third option is a second verification loop: the new model computes the feature in parallel with the running one, and the decision is made by agreement of both.
This reduces the false pass rate where the cost of error is high, but adds load to computational resources and increases decision time, so only critical access points are handled this way.
The transfer of developments is the same for any path: labeling, verification scenarios, event history, and integrations with the accounting system are reused — only what is tied to the features of a specific model is rebuilt.
We fix the choice of path after measuring on accumulated history: we compare the false pass rates and false trigger rates for each scheme on the site's real cameras.
The update proceeds without stopping operations — the previous model remains in service until the new one confirms the result on the same stream.
Common technical questions before starting
Before starting ArcFace training, customers raise the same set of technical concerns: whether there is enough data, whether it is legal to store biometrics, what happens when cameras are changed, and how the model will behave at multiple points.
The data collection architecture depends on the answers, so we address them before the first line of code — some limitations are later removed by adjusting the shooting area, some are not removed at all.
Is there enough data and what to do with rare angles?
We aim for dozens of frames per person under different conditions — light, head rotation, glasses, headwear. We collect rare angles from working cameras and add them to the sample: without this, false denials fall specifically on them.
How is biometric data stored and is employee consent required?
The database stores not a photograph but a numerical feature vector — an image cannot be reconstructed from it. Consent is formalized in a separate document, access is differentiated by roles, we keep a log of requests, and delete records upon request.
What happens when cameras are changed, and does quality degrade over time?
The model is sensitive to the scene: angle, height, lighting, distance. After equipment replacement, we re-select thresholds, and then monitor the denial rate — new people arrive, lighting and seasonal clothing change, so we periodically fine-tune the model on an updated sample.
How does the system scale to multiple points?
Comparison is done by index, not brute force, so response time does not grow linearly with the size of the database. We connect points to a common database, fix thresholds separately for each entrance, and verify the false trigger rate.
Let's discuss your face recognition task
To come back with a project estimate, work plan, and timelines instead of general words, we need a small set of initial data — and the closer they are to the real conditions of the site, the more accurate the forecast for model quality. Especially when it comes to training ArcFace for your scenario.
- Examples of data from the site: 30–50 images or short clips from the same cameras, under the same lighting and angles.
- Composition of people: how many people to distinguish, how many frames per person already exist or can be collected.
- Scenario: access control, time tracking, archive search — the threshold and denial logic depend on this.
- Quality requirements: acceptable false pass and false trigger rates, response to glasses, masks, headwear.
- Data constraints: where images are stored, who gives consent, what anonymization rules apply.
- Infrastructure: camera stream or offline processing, computational base, integration with existing video analytics.
- Stage boundaries: a pilot site or all facilities at once, critical deadlines, who accepts the result.
After discussing the task, we will return with a project estimate, a step-by-step work plan, and a proposal for a trial run: training and validation on your data, before scaling to the entire infrastructure.
