Custom NER Model Training: Turnkey Solution

You're working with domain texts—medical, legal, financial. Standard NER models (e.g., spaCy or Stanford NER) miss up to 30% of specific entities: license numbers, drug codes, references to legislative acts. A custom TokenClassification model solves the problem—we train it with F1 up to 97%. Our NER model training service starts at €3,500, and clients typically save over $5,000 in data correction costs. With extensive experience in NLP and over 30 projects, we've learned to navigate common pitfalls such as unbalanced annotation and label alignment.

Named Entity Recognition (NER) is the task of extracting named entities. Fine-tuning NER on custom types is standard practice, but there are many pitfalls: from IOB2 annotation errors to incorrect loss calculation on special tokens. We offer expert fine-tuning NER services for domain adaptation.

What Problems We Solve

• Chaotic annotation: IOB2 tags with omissions and overlaps reduce F1 by 10–15%. We audit, fix inconsistencies, and standardize the format. One client saved over 200,000 rubles on error correction after our audit.
• Rare entities: The "License Number" type may appear 20 times in 10,000 documents. Without oversampling or augmentation, the model simply won't learn it. We use entity replacement from a predefined dictionary—boosting F1 by up to 8%.
• Model selection: DeepPavlov/rubert-base-cased gives 94% on PER but may drop to 70% on custom legal entities. We test 3–4 pre-trained models (ruBERT, XLM-R, mBERT) and pick the best for your domain. DeepPavlov/rubert is 1.5× more accurate than mBERT on Russian custom entities. Our NLP model outperforms default spaCy NER by 3× on legal entity extraction.

[CLS] Patient [B-PER] Ivanov [I-PER] complains [O] of [O] headache [B-SYMPTOM] [I-SYMPTOM] . [O] Diagnosis [O] : [O] migraine [B-DIAGNOSIS] . [O] Prescribed [O] paracetamol [B-DRUG] . [SEP] 

How We Do It

Consider a case: medical protocols where entities "Symptom", "Diagnosis", and "Drug" need extraction. Initial annotation—500 sentences in IOB2. Our entity extraction pipeline:

1. Fix errors: in 10% of sentences, confusion between B-Diagnosis and I-Diagnosis.
2. Augment: replace drug names from a dictionary of analogues, expanding the sample to 1500 sentences.
3. Fine-tune on DeepPavlov/rubert-base-cased: 10 epochs, batch size 16, learning rate 5e-5.
4. Evaluate using seqeval: F1 per entity—symptom 91%, diagnosis 94%, drug 96%.

Fine-tuning process for TokenClassification:

from transformers import AutoModelForTokenClassification, TrainingArguments, Trainer from transformers import DataCollatorForTokenClassification label_list = ["O", "B-PER", "I-PER", "B-ORG", "I-ORG", "B-LOC", "I-LOC"] label2id = {l: i for i, l in enumerate(label_list)} id2label = {i: l for l, i in label2id.items()} model = AutoModelForTokenClassification.from_pretrained( "DeepPavlov/rubert-base-cased", num_labels=len(label_list), id2label=id2label, label2id=label2id ) data_collator = DataCollatorForTokenClassification(tokenizer) training_args = TrainingArguments( output_dir="./ner_model", num_train_epochs=10, # NER requires more epochs than classification per_device_train_batch_size=16, learning_rate=5e-5, weight_decay=0.01, ) 

NER metrics: seqeval library (BIO format)

import evaluate seqeval = evaluate.load("seqeval") def compute_metrics(p): predictions, labels = p predictions = np.argmax(predictions, axis=2) true_predictions = [ [label_list[p] for (p, l) in zip(pred, label) if l != -100] for pred, label in zip(predictions, labels) ] true_labels = [ [label_list[l] for (p, l) in zip(pred, label) if l != -100] for pred, label in zip(predictions, labels) ] results = seqeval.compute(predictions=true_predictions, references=true_labels) return {"f1": results["overall_f1"], "precision": results["overall_precision"]} 

Process of Work

1. Analytics: Study your domain, entity list, data volume.
2. Audit & annotation preparation: Check IOB2, augment, fix errors.
3. Model selection: Test 3 architectures on a validation set.
4. Fine-tuning with hyperparameters: learning rate, epochs, scheduling.
5. Testing: On a held-out set with per-type evaluation.
6. Deployment: Conversion to ONNX, integration via REST API or library.

What's Included

• Final report with metrics per entity type.
• Fine-tuning and inference code (Jupyter notebook or Python script).
• Model in ONNX or TorchScript format for production.
• Brief documentation on parameters and supported entities.
• 2 weeks of support after delivery—we answer questions on further tuning.

How to Choose a Model Architecture?

DeepPavlov/rubert gives the best F1 on Russian for standard entities, but for custom ones with small data, XLM-RoBERTa may be more stable. For domain adaptation, we test multiple pretrained models.

Comparison of Augmentation Methods

Why Quality Annotation Matters

IOB2 annotation is key. Errors like missing B-tag or entity overlap reduce final F1 by 10–20%. We guarantee that after our annotation audit and fix, you'll gain at least 5% accuracy improvement with the same data volume. A robust NLP model relies on precise token alignment.

Common Fine-Tuning Mistakes

• Too few epochs: 3 epochs is insufficient for NER—need 8–12.
• Unbalanced batch: if one entity type appears in 90% of sentences, use weighted loss or undersampling.
• Ignoring special tokens: don't forget to mask [CLS], [SEP], and padding when calculating loss.

Metrics and Guarantee

We rely on seqeval for transparent evaluation. Typical F1 after our tuning: PER 94–97%, ORG 88–93%, custom domain entities 80–92%. We guarantee stable metrics on the test set: if F1 falls 5% or more below the stated threshold, we fine-tune for free. Our medical NLP solutions are tailored for clinical texts.

Contact us to assess your case—we'll recommend the optimal architecture and annotation volume. Order your NER model training today.