Optimize ML Models for Mobile via Knowledge Distillation

A mobile neural network must be lightweight yet accurate. Knowledge Distillation is a method that compresses a large model without critical quality loss. We have applied this approach on dozens of projects: from image classification to on-device NLP. The result — a model 10 times smaller with only a 1–2% accuracy drop. This approach is especially in demand in mobile ML, where every megabyte and millisecond count. Contact us — we will audit your model and propose a distillation plan.

Why Soft Labels Work Better

Standard training: correct class = 1.0, others = 0.0. Hard labels. A teacher on an image of a cat outputs: cat 0.85, lynx 0.08, tiger 0.04, dog 0.02 ... These "soft" labels carry information that a lynx is more similar to a cat than to an airplane. A student trained on such labels learns the structure of feature space, not just a binary classifier decision. This is the essence of the student-teacher paradigm.

import torch import torch.nn.functional as F def distillation_loss(student_logits, teacher_logits, true_labels, temperature=4.0, alpha=0.7): """ alpha — weight of distillation vs hard label loss temperature — smooths teacher distribution """ # Soft targets loss (KL divergence between student and teacher) soft_teacher = F.softmax(teacher_logits / temperature, dim=-1) soft_student = F.log_softmax(student_logits / temperature, dim=-1) distill_loss = F.kl_div(soft_student, soft_teacher, reduction='batchmean') * (temperature ** 2) # Hard label loss (standard cross-entropy) hard_loss = F.cross_entropy(student_logits, true_labels) return alpha * distill_loss + (1 - alpha) * hard_loss 

temperature ** 2 — normalizing factor that compensates for gradient scale at high temperature. Without it, distill_loss and hard_loss are at different scales.

Choosing Student Architecture

The student must be smaller than the teacher, but not arbitrarily. Good base architectures for mobile:

• MobileNetV3-Small — 2.5 MB, designed for mobile from scratch, depthwise separable convolutions
• EfficientNet-Lite0/1 — good accuracy/speed balance
• MobileViT-XXS — hybrid CNN+Transformer, 1.3 MB
• DistilBERT (for NLP) — already distilled from BERT, 66 MB vs 440 MB

For object detection on mobile: student based on YOLOv8n (8 MB) distilled from YOLOv8l (87 MB).

How Distillation Methods Compare?

There are several distillation approaches, differing in complexity and outcome. Main ones: logit distillation (on outputs), intermediate layer distillation, and data-free distillation. Choice depends on data availability and accuracy requirements.

Distillation Steps

1. Teacher preparation: fine-tune or quantize existing model, freeze weights.
2. Student architecture selection: fit resource budget (RAM, CPU, latency).
3. Hyperparameter tuning: temperature (usually 2-8), alpha (0.5-0.9), choice of layers for intermediate distillation.
4. Student training: minimize combined loss, monitor validation accuracy.
5. Conversion to Core ML/TFLite with int8 quantization for final size.
6. Validation on real devices: check latency and accuracy.

Distillation Process: Example for Classification

# Assume: teacher — ResNet-50, student — MobileNetV3-Small teacher = torchvision.models.resnet50(pretrained=True).eval() student = torchvision.models.mobilenet_v3_small(pretrained=False) # Freeze teacher for param in teacher.parameters(): param.requires_grad = False optimizer = torch.optim.AdamW(student.parameters(), lr=1e-3, weight_decay=1e-4) scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=100) for epoch in range(100): student.train() for images, labels in train_loader: with torch.no_grad(): teacher_logits = teacher(images) student_logits = student(images) loss = distillation_loss(student_logits, teacher_logits, labels, temperature=4.0, alpha=0.7) optimizer.zero_grad() loss.backward() optimizer.step() scheduler.step() val_acc = evaluate(student, val_loader) print(f"Epoch {epoch}: student_acc={val_acc:.4f}") 

Typical results: MobileNetV3-Small trained normally — 67–68% top-1 on ImageNet. After distillation from ResNet-50 — 71–72%. Increase of 3–4% due to knowledge transfer.

How Does Intermediate Layer Distillation Work?

Distillation only on outputs (logits) is the basic version. A stronger version: add correspondence of intermediate feature maps.

# FitNets / PKT: student learns teacher's feature maps class DistillationHook: """Hook to capture intermediate activations""" def __init__(self): self.output = None def __call__(self, module, input, output): self.output = output teacher_hook = DistillationHook() student_hook = DistillationHook() # Register on corresponding layers teacher.layer3.register_forward_hook(teacher_hook) student.features[9].register_forward_hook(student_hook) # Analogous layer # In training loop add feature distillation loss with torch.no_grad(): teacher(images) teacher_features = teacher_hook.output student(images) # with grad student_features = student_hook.output # If dimensions differ — need adapter (1x1 Conv) if teacher_features.shape[1] != student_features.shape[1]: student_features = adapter_conv(student_features) # adapter trained jointly feature_loss = F.mse_loss(student_features, teacher_features.detach()) 

This approach requires alignment of feature map dimensions between teacher and student — via 1×1 convolutional adapters. The adapter adds a few parameters to the student but remains small.

How to Distill a Model if Original Data Is Unavailable?

Sometimes the original dataset is unavailable (IP restrictions, privacy). Data-free distillation — generate synthetic data that maximizes teacher activations:

# DAFL (Data-Free Learning): generator creates "samples" for distillation generator = Generator(latent_dim=256, img_channels=3) optimizer_G = torch.optim.Adam(generator.parameters(), lr=1e-4) for step in range(1000): z = torch.randn(batch_size, 256) fake_images = generator(z) # Losses: maximize teacher confidence + minimize BatchNorm statistics mismatch teacher_out = teacher(fake_images) activation_loss = -teacher_out.max(dim=1)[0].mean() # teacher should be confident # BN statistics matching bn_loss = compute_bn_statistics_loss(teacher, fake_images) total_loss = activation_loss + 0.1 * bn_loss optimizer_G.zero_grad() total_loss.backward() optimizer_G.step() 

The quality of data-free distillation is lower than the full-data version, but sometimes it is the only option.

Distillation for NLP Tasks on Mobile

For mobile apps with NLP (sentiment classification, intent detection, summarization): distill from GPT-4 / Claude API responses into a small BERT/DistilBERT.

# Collect soft labels from teacher (GPT-4 API) # For each training example request class probabilities # Store as training labels for student # Student — DistilBERT fine-tuned on these soft labels 

DistilBERT (66 MB, ONNX int8 — 18 MB) runs on-device in 30–80 ms on iOS/Android. GPT-4 in the cloud — hundreds of ms, costs per request.

What's Included in Our Work

• Resource budget analysis: RAM, CPU, GPU, power consumption
• Student architecture selection based on your constraints
• Teacher preparation (fine-tuning, quantization)
• Distillation with hyperparameter tuning (temperature, alpha, intermediate layers)
• Validation on test set and on real devices
• Conversion to Core ML / TFLite with int8 quantization
• Integration and support (updates for new OS versions)

Our team has 10+ years of experience in mobile development and ML. We guarantee the model will run stably on iOS 14+ and Android 10+. Get a free engineer consultation on your model. Order distillation — we'll prepare a preliminary estimate in 1 day.

Timeline Estimates

Basic logit distillation for a classification task — 2–4 weeks (GPU time plus hyperparameter tuning). Full distillation with intermediate layers, non-standard architectures, data augmentation — 5–10 weeks. Custom estimate upon request.

Further reading: Knowledge Distillation on Wikipedia, PyTorch tutorial.