Why data annotation in Prodigy runs into manual labor
Data annotation in Prodigy is built around one example at a time: the interface shows text, a model suggestion, and waits for a label.
The tool carefully records the answer, but it does not decide which examples should be shown at all — the order is set by a human or random sampling.
Hence the first annotation bottleneck: a significant share of the output consists of obvious cases where the label matches the model prediction and contributes nothing to training.
Annotators spend most of their time on them, while ambiguous examples receive exactly one label and go into the dataset without rechecking.
The second failure is label disagreement between annotators. Working blindly, each relies on their own reading of the instructions, so opposite answers appear on borderline examples.
Annotation agreement drops, and there is no way to check it: there is neither a second pass nor conflict resolution.
As a result, training set quality is determined not by the task but by who annotated it and in what order.
The model keeps training on ambiguous examples and memorizes noise — quality on the holdout set stagnates, while the volume of manual dataset annotation grows along with accuracy requirements.
We break down how this is addressed: example prioritization, agreement control, and re-annotation of borderline cases.
What training a model on ambiguous examples gives
Active learning flips the annotation order: the annotator no longer goes through the corpus sequentially but receives only the records where the model is uncertain.
The queue is formed by uncertainty — by the probability gap between the two leading classes and by distribution entropy. Examples with the smallest margin go to a human first, while confident predictions await selective review.
In Prodigy, the loop is closed: the label immediately enters the training set, the model is further trained, and the queue for the next batch is recalculated.
The project does not wait until the entire corpus is annotated — quality improves with every hundred examples, and from the first passes it is clear which classes are missing.
The effect is measurable. The share of records that truly require manual review drops by a factor of 3–5 compared with exhaustive annotation: 80–90% of the corpus consists of unambiguous examples, and they go into semi-automatic annotation.
Accuracy on ambiguous slices is higher than with uniform sampling: each ambiguous record brings the model the maximum amount of new information.
Selecting ambiguous examples also exposes defects in the schema itself: contradictory labels on similar records, overlapping classes, rare categories without a single confident example.
We examine such cases before release — we transfer them into the annotator instructions and validation checks, rather than discovering them in production through user complaints.
Annotation modes: manual, semi-automatic, hybrid
The annotation mode is chosen based on the nature of the data and the cost of an error, not on the convenience of the performer.
One scheme produces a gold standard on a small volume, another accelerates throughput on homogeneous classes, and a third combines model pre-annotation with human correction — and each has conditions under which it starts working against the project.
| Mode | When it fits | Limitations |
|---|---|---|
| Controlled manual annotation | Small sample, high cost of error, project start, and instruction calibration | Time grows linearly with volume; by the end of a shift, annotators' interpretations diverge |
| Semi-automatic (active learning) | Homogeneous classes, thousands of examples, an initial sample for model initialization exists | The loop needs restarting when classes drift, and labels need revisiting when the schema changes |
| Hybrid | Large, sparsely annotated streams, several performers, annotation in the product stream | The model error and human error must be separated; more expensive to configure selection rules |
The hybrid mode is usually adopted on long projects: the model suggests an option, a human confirms or rejects it, and ambiguous examples go into a separate queue for rechecking.
The volume of pure manual work drops, but without an agreement metric between annotators, noise in the training set remains unnoticed — and shows up in model quality.
The guideline when choosing is the share of ambiguous examples: if it is above one-third, the semi-automatic mode starts suggesting the human's own errors back to them, and the loop becomes self-referential.
Prodigy allows all three modes to be run in one interface and switched as the sample grows, without losing already annotated data.
How we build the annotation loop for the task
We begin annotation implementation not with launching the tool but with an audit of the label schema: we examine the current corpus, find overlapping and “dead” classes, and estimate how many examples per class actually exist.
A schema with 40 labels and three examples per label will not yield a trainable model either with active learning or manual sampling — so we merge some classes and convert others into attributes.
Then we assemble the annotation pipeline, and only after that do we put people to annotation; otherwise, we would have to redo an already completed set.
- Audit of the label schema and corpus. We check the agreement of the current annotation, calculate the share of ambiguous examples and class imbalance. The output is a reduced schema with instructions for borderline cases.
- Design of the annotation pipeline. We define the interface for the task type — classification, entities, relations — annotator roles, the share of examples for cross-checking, and the conflict resolution procedure.
- Active learning setup. We connect a candidate model on the current Prodigy branch: it selects low-confidence examples and ambiguous cases into the queue, not a random sample.
- Pilot on a control batch. We annotate a small volume, measure agreement between annotators, and revise the instructions before annotation goes into production volume.
- Training and quality measurement. We prepare a spaCy model, compare metrics on the holdout set, and see which error classes grow when new data is added.
- Annotation policy. We fix the label schema, quality criteria, escalation procedure for ambiguous cases, and rules for versioning datasets.
- Handover and support. We show the client's team the queue, metrics, and retraining points; we separately explain how to change the label schema without losing the already annotated corpus.
The pipeline remains with the client — with the queue, quality dashboard, and a clear policy, so that annotation does not turn into a one-off project.
Delivery: configuration, label schema, review policy
We hand over a configured Prodigy project as a working kit, not as an archive of files: the client's team should be able to run annotation and model retraining without us. Below is exactly what remains in the repository and in the annotation documentation.
- Project configuration: annotation interface for the task, thresholds for selecting examples for active learning, integration with the spaCy model, and its retraining parameters.
- Label schema: complete list of classes, rules for mutual exclusivity and priorities, examples for each class, and separately — borderline cases where annotators most often disagree.
- Annotator instructions: length of selected fragments, actions with ambiguous and nested entities, and the issue escalation procedure.
- Annotation review policy: share of tasks for re-annotation, discrepancy review, agreement threshold, and the person responsible for the final decision.
- Annotation quality criteria: inter-annotator agreement metrics, share of rejected tasks, and the condition for returning the model to a new training iteration.
- Gold standard example set: a fixed sample on which we run every new model version and compare with the previous one.
- Data flow scheme: how annotated examples enter training, how the dataset is versioned, and how to roll back to a previous version.
We hand over everything with team training and brief notes on each artifact — so that changes to the label schema are made deliberately. Let's discuss the task — we will come back with a delivery plan for your process.
Case study: ticket classification and growing label agreement
What it was before
Annotation was done in spreadsheets: several annotators entered ticket categories into a shared sheet, and ambiguous cases were sorted out by manual review once a week.
Class rules lived in chat threads and in each person's head, so the same request “payment does not go through” landed sometimes in “payment” and sometimes in “integration failure.” Annotator agreement hovered around 0.6 by share of matching labels — the model learned from contradictory examples and fell apart precisely on borderline classes.
What we changed and what the result was
We assembled a unified annotation pipeline in Prodigy: a taxonomy of 14 classes, hotkey input, and a mandatory comment on an ambiguous label.
We enabled Active Learning — the model selects low-confidence examples for annotation rather than random ones, so annotator time goes to borderline cases.
Categories where people disagreed more often are annotated by two people, and discrepancies automatically fall into a review queue.
Annotation quality metrics became part of the process, not a post hoc report: annotator agreement rose to 0.9, the share of ambiguous cases dropped by about a factor of three, and time per ticket fell severalfold due to model pre-annotation.
Ticket classification on the holdout set improved noticeably, while manual reviews left the regular cycle — retraining became less frequent, and errors on borderline classes are handled where they appear.
What to do if the model is already trained on noisy labels?
A model trained on noisy labels almost always overestimates itself on validation: it learns not the class feature but the errors of a particular annotator.
If the noise is spread across the sample, the holdout part contains the same errors, and the metric looks acceptable until data starts coming from the real stream.
Therefore the first step is not retraining but diagnostics: we calculate the model's confidence on the training set and look for where it diverges from the assigned label.
The discrepancy falls into two fundamentally different categories. Where the model is confident in one class but the label is on the opposite one, this is highly likely a noisy label — we send it for human rechecking.
Where confidence stays near the threshold, it is more likely about borderline classes: the example objectively lies at the boundary of two categories, and the fault is not the annotator's quality but an unclear rule. We do not delete such examples — we align the instructions and record the decision in the schema.
The recheck queue is not built randomly: in the production pipeline on Prodigy, we prioritize specifically model-label discrepancies, from the most confident to the ambiguous.
Each correction preserves the original value, the reviewer's verdict, and the reason — already collected data remains in the annotation database, and nothing is overwritten blindly.
Next comes layer-by-layer annotation cleaning: ambiguous rows go into quarantine, the agreed layer becomes the training one, and rule versions are fixed so that retraining is reproducible.
As a result, the holdout metric stops being inflated, and borderline classes receive separate rules instead of averaged noise.
Solution limitations and common technical concerns
The annotation pipeline in Prodigy runs into the tool's boundaries, not its settings: active learning speeds up only those classes that the model already partially knows. Below are the limitations we record at the start and the engineering ways to work around them.
How do several annotators work simultaneously without duplicating each other?
Tasks come from a shared queue, and each example is assigned to one performer — parallel work without overlaps, and answers are not lost if the connection drops.
We annotate part of the sample in an overlapping way: we calculate agreement by the share of matches and correct the instructions, not individual errors.
What to do if a rare class does not accumulate enough examples?
The suggestion model selects only what it already knows, so for rare classes we disable it and search for candidates with regexes, dictionaries, or vector similarity. We lower the confidence threshold, and extra matches go to review.
Can annotation done earlier in other tools be migrated?
Yes: we convert familiar formats — spreadsheets, JSON, entity annotations in text — into the required schema while preserving character offsets; otherwise entity boundaries will shift.
Discrepancies in rules (nested entities, different interpretations of boundaries) are handled separately.
When to retrain the model, and how to notice that it got worse?
The cycle: an annotation batch, retraining, a run on the frozen control batch, and selecting errors for the next batch. A set of a couple hundred examples with double annotation provides precision and recall by class — regressions are visible before rollout.
Let's discuss the task: what to send and how the work will proceed
To come back with an annotation project estimate rather than guesses, we need a minimum of context from you. Annotation almost always takes more time than model training, so we agree on the label schema and quality requirements before the first annotator opens the first record.
- A sample of data “as is” — several hundred real records with noise, typos, and incomplete fields: we use them to check that the label schema withstands your domain, not just clean examples.
- A draft of what the model should distinguish: a list of entities or classes and 10–15 cases each of “this is correct / this is incorrect.” Ambiguous cases are more valuable here than unambiguous ones.
- Source and volume: where we take the array from, what form it is in now, how many records are planned for annotation, and how often new ones arrive.
- Data restrictions: personal information, what cannot be exported outside the perimeter, requirements for storage and deletion of copies.
- Acceptance criterion: what share of matches with the gold standard is considered sufficient and on which scenario this is checked.
- Current process, if one exists: who annotates, in which files records are kept, and how much time one record takes.
- Domain glossary and data language: terms, abbreviations, language mixing — they will need to be fixed in the annotator instructions.
Then the work proceeds as follows: we audit the submitted array, come back with an annotation project estimate and an annotation implementation plan — label schema, volume, iteration order, and quality control points.
Before scaling, we run a pilot on several hundred examples and calculate annotator agreement: if the instructions allow for different interpretations, this is visible before the entire array is annotated.
We annotate in Prodigy, where active learning pulls the most ambiguous examples into the queue, so the share of manual work drops as the project progresses. Describe the annotation task — we will come back with an estimate and a plan.
