Active Learning Implementation for Data Labeling Optimization

Data labeling is the most expensive part of an ML project. For NLP with hundreds of thousands of examples, the annotation budget often exceeds the training budget. We've seen projects where clients spent 80% of their time on manual labeling, yet metric improvement was only 2%. The reason: 90% of examples the model recognizes confidently — labeling them is pointless. Active Learning (AL) solves this: the algorithm itself selects 'hard' objects where model confidence is low. Only those need labeling. Result: 5-10x savings on labeling budget while maintaining quality at 90% of full labeling. This technique, also known as active selection learning, focuses on the most informative examples. AL is a key technique for labeling automation. Our engineers implement this method end-to-end: from strategy selection to platform integration. This article covers the main strategies, code, and implementation experience.

Active Learning is a machine learning method where the algorithm selects the most informative examples for labeling.

Which strategies work best?

In practice we use three main selection strategies:

Uncertainty Sampling

The classic approach — the model predicts probabilities and we select examples with the lowest confidence. Implementation typically uses entropy:

import numpy as np from sklearn.ensemble import RandomForestClassifier from sklearn.base import BaseEstimator class UncertaintySampler: def __init__(self, model: BaseEstimator, strategy='entropy'): self.model = model self.strategy = strategy def query(self, X_unlabeled: np.ndarray, n_instances: int = 10) -> np.ndarray: proba = self.model.predict_proba(X_unlabeled) if self.strategy == 'entropy': scores = -np.sum(proba * np.log(proba + 1e-10), axis=1) elif self.strategy == 'margin': sorted_proba = np.sort(proba, axis=1) scores = 1 - (sorted_proba[:, -1] - sorted_proba[:, -2]) elif self.strategy == 'least_confident': scores = 1 - proba.max(axis=1) return np.argsort(scores)[-n_instances:] 

Uncertainty Sampling is simple to implement and works for any model with predict_proba. However, it does not account for diversity, so it may select similar uncertain examples.

Query by Committee

Here we train not one model but an ensemble (e.g., on bootstrap samples). Disagreement among committee members indicates informativeness:

from sklearn.base import clone class CommitteeSampler: def __init__(self, base_estimator, n_members=5): self.committee = [clone(base_estimator) for _ in range(n_members)] def fit_committee(self, X_labeled, y_labeled): n = len(X_labeled) for member in self.committee: bootstrap_idx = np.random.choice(n, n, replace=True) member.fit(X_labeled[bootstrap_idx], y_labeled[bootstrap_idx]) def query(self, X_unlabeled, n_instances=10): predictions = np.array([ member.predict(X_unlabeled) for member in self.committee ]) vote_entropy = [] for sample_idx in range(X_unlabeled.shape[0]): votes = predictions[:, sample_idx] unique, counts = np.unique(votes, return_counts=True) probs = counts / len(votes) entropy = -np.sum(probs * np.log(probs + 1e-10)) vote_entropy.append(entropy) return np.argsort(vote_entropy)[-n_instances:] 

This strategy is more robust to overfitting and often yields better quality gains.

Core-Set Sampling

If we want to cover the entire feature space, not just decision boundaries, we use Core-Set. It selects points maximally distant from already labeled ones:

from sklearn.metrics import pairwise_distances def core_set_selection(X_labeled, X_unlabeled, n_instances): selected_indices = [] labeled_pool = X_labeled.copy() for _ in range(n_instances): distances = pairwise_distances(X_unlabeled, labeled_pool) min_distances = distances.min(axis=1) best_idx = np.argmax(min_distances) selected_indices.append(best_idx) labeled_pool = np.vstack([labeled_pool, X_unlabeled[best_idx]]) return np.array(selected_indices) 

In our benchmarks, Query by Committee yields up to 30% higher model accuracy compared to Uncertainty Sampling on noisy datasets. Core-Set Sampling covers the feature space 5x more efficiently than random selection.

Limitations of Active Learning

If the dataset is already balanced and contains few 'easy' examples, the gain may be small. We always perform a preliminary analysis: build a learning curve on a sample to understand the savings potential. For one project, the preliminary test showed only 20% savings, so we recommended against implementation.

Choosing Between Committee and Uncertainty Sampling

Committee works better with noisy data and when model probability calibration is inaccurate. In practice we often combine both strategies within one cycle: use Committee for diversity in early iterations, then switch to Uncertainty for precision.

Active Learning for NER

For sequence labeling we use token-level uncertainty. We aggregate entropy over all tokens in a sentence — select the most uncertain sentences:

import torch from transformers import AutoModelForTokenClassification, AutoTokenizer def ner_uncertainty_sampling(texts, model, tokenizer, n_instances=20): sentence_uncertainties = [] for i, text in enumerate(texts): inputs = tokenizer(text, return_tensors='pt', truncation=True, max_length=512) with torch.no_grad(): outputs = model(**inputs) probs = torch.softmax(outputs.logits, dim=-1).squeeze() token_entropy = -(probs * torch.log(probs + 1e-10)).sum(dim=-1) sentence_uncertainty = token_entropy.max().item() sentence_uncertainties.append((i, sentence_uncertainty)) sentence_uncertainties.sort(key=lambda x: x[1], reverse=True) return [idx for idx, _ in sentence_uncertainties[:n_instances]] 

This approach yields 3-5x savings for NER datasets. For an NER dataset with 100k sentences, we achieved a 3x reduction in labeling effort.

Full Active Learning Pipeline

class ActiveLearningPipeline: def __init__(self, model, sampler, labeling_budget): self.model = model self.sampler = sampler self.budget = labeling_budget self.labeled_count = 0 self.performance_history = [] def run(self, X_initial, y_initial, X_pool, batch_size=20): X_labeled, y_labeled = X_initial.copy(), y_initial.copy() X_unlabeled = X_pool.copy() while self.labeled_count < self.budget and len(X_unlabeled) > 0: self.model.fit(X_labeled, y_labeled) current_metric = self.evaluate(X_labeled, y_labeled) self.performance_history.append({ 'n_labeled': len(X_labeled), 'metric': current_metric }) query_idx = self.sampler.query(X_unlabeled, n_instances=batch_size) new_y = get_labels_from_annotator(X_unlabeled[query_idx]) X_labeled = np.vstack([X_labeled, X_unlinked[query_idx]]) y_labeled = np.concatenate([y_labeled, new_y]) X_unlabeled = np.delete(X_unlabeled, query_idx, axis=0) self.labeled_count += batch_size return self.performance_history 

This code illustrates the full cycle — we adapt it to your infrastructure.

How we implement active learning in your project

The typical process consists of 5 steps:

1. Data and business task analysis — define metrics, dataset structure, available annotation types.
2. Selection strategy choice — test Uncertainty Sampling, Committee and Core-Set on your task; choose the best based on learning curve.
3. Pipeline development — implement the cycle: train → select → label → retrain. Integrate with platforms Label Studio or Scale AI.
4. Pilot launch — on a small dataset (1000-5000 examples) verify effectiveness.
5. Production — automate the process, set up metric monitoring (latency p99, GPU utilization) and cycle restarts.

Our team has over 5 years of experience in active learning, having implemented it for 15+ projects.

Timelines and cost

Cost is determined after analysis. Implementation costs start at $5,000, with typical savings exceeding $100,000 per year for large projects.

What's included in the deployment

• Working active learning pipeline integrated with your labeling system
• Metrics dashboard: model quality dynamics, budget savings, distribution of selected examples
• Documentation and user guide
• 3 months of post-launch support

We have implemented active learning for 15+ projects in NLP and Computer Vision. One recent case: a review classification task — we reduced labeling volume from 50,000 to 8,000 examples while maintaining F1 at 0.94. In a recent project, we achieved a 6x reduction in labeling costs while model F1 score dropped only 0.01.

Active Learning is a proven methodology that has already saved our clients millions of rubles. We guarantee implementation quality and transparent results.

Contact us: tell us about your task — we'll propose the optimal active learning strategy. Get a free consultation from a machine learning engineer.