Optimize ML Models for Mobile via Knowledge Distillation

A mobile neural network must be lightweight yet accurate. Knowledge Distillation is a method that compresses a large model without critical quality loss. We have applied this approach on dozens of projects: from image classification to on-device NLP. The result — a model 10 times smaller with only a 1–2% accuracy drop. This approach is especially in demand in mobile ML, where every megabyte and millisecond count. Contact us — we will audit your model and propose a distillation plan.

Why Soft Labels Work Better

Standard training: correct class = 1.0, others = 0.0. Hard labels. A teacher on an image of a cat outputs: cat 0.85, lynx 0.08, tiger 0.04, dog 0.02 ... These "soft" labels carry information that a lynx is more similar to a cat than to an airplane. A student trained on such labels learns the structure of feature space, not just a binary classifier decision. This is the essence of the student-teacher paradigm.

import torch
import torch.nn.functional as F

def distillation_loss(student_logits, teacher_logits, true_labels, temperature=4.0, alpha=0.7):
    """
    alpha — weight of distillation vs hard label loss
    temperature — smooths teacher distribution
    """
    # Soft targets loss (KL divergence between student and teacher)
    soft_teacher = F.softmax(teacher_logits / temperature, dim=-1)
    soft_student = F.log_softmax(student_logits / temperature, dim=-1)
    distill_loss = F.kl_div(soft_student, soft_teacher, reduction='batchmean') * (temperature ** 2)

    # Hard label loss (standard cross-entropy)
    hard_loss = F.cross_entropy(student_logits, true_labels)

    return alpha * distill_loss + (1 - alpha) * hard_loss

temperature ** 2 — normalizing factor that compensates for gradient scale at high temperature. Without it, distill_loss and hard_loss are at different scales.

Choosing Student Architecture

The student must be smaller than the teacher, but not arbitrarily. Good base architectures for mobile:

• MobileNetV3-Small — 2.5 MB, designed for mobile from scratch, depthwise separable convolutions
• EfficientNet-Lite0/1 — good accuracy/speed balance
• MobileViT-XXS — hybrid CNN+Transformer, 1.3 MB
• DistilBERT (for NLP) — already distilled from BERT, 66 MB vs 440 MB

For object detection on mobile: student based on YOLOv8n (8 MB) distilled from YOLOv8l (87 MB).

How Distillation Methods Compare?

There are several distillation approaches, differing in complexity and outcome. Main ones: logit distillation (on outputs), intermediate layer distillation, and data-free distillation. Choice depends on data availability and accuracy requirements.

Distillation Steps

1. Teacher preparation: fine-tune or quantize existing model, freeze weights.
2. Student architecture selection: fit resource budget (RAM, CPU, latency).
3. Hyperparameter tuning: temperature (usually 2-8), alpha (0.5-0.9), choice of layers for intermediate distillation.
4. Student training: minimize combined loss, monitor validation accuracy.
5. Conversion to Core ML/TFLite with int8 quantization for final size.
6. Validation on real devices: check latency and accuracy.

Distillation Process: Example for Classification

# Assume: teacher — ResNet-50, student — MobileNetV3-Small
teacher = torchvision.models.resnet50(pretrained=True).eval()
student = torchvision.models.mobilenet_v3_small(pretrained=False)

# Freeze teacher
for param in teacher.parameters():
    param.requires_grad = False

optimizer = torch.optim.AdamW(student.parameters(), lr=1e-3, weight_decay=1e-4)
scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=100)

for epoch in range(100):
    student.train()
    for images, labels in train_loader:
        with torch.no_grad():
            teacher_logits = teacher(images)
        student_logits = student(images)
        loss = distillation_loss(student_logits, teacher_logits, labels, temperature=4.0, alpha=0.7)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
    scheduler.step()
    val_acc = evaluate(student, val_loader)
    print(f"Epoch {epoch}: student_acc={val_acc:.4f}")

Typical results: MobileNetV3-Small trained normally — 67–68% top-1 on ImageNet. After distillation from ResNet-50 — 71–72%. Increase of 3–4% due to knowledge transfer.

How Does Intermediate Layer Distillation Work?

Distillation only on outputs (logits) is the basic version. A stronger version: add correspondence of intermediate feature maps.

# FitNets / PKT: student learns teacher's feature maps
class DistillationHook:
    """Hook to capture intermediate activations"""
    def __init__(self):
        self.output = None
    def __call__(self, module, input, output):
        self.output = output

teacher_hook = DistillationHook()
student_hook = DistillationHook()

# Register on corresponding layers
teacher.layer3.register_forward_hook(teacher_hook)
student.features[9].register_forward_hook(student_hook)  # Analogous layer

# In training loop add feature distillation loss
with torch.no_grad():
    teacher(images)
teacher_features = teacher_hook.output
student(images)  # with grad
student_features = student_hook.output

# If dimensions differ — need adapter (1x1 Conv)
if teacher_features.shape[1] != student_features.shape[1]:
    student_features = adapter_conv(student_features)  # adapter trained jointly
feature_loss = F.mse_loss(student_features, teacher_features.detach())

This approach requires alignment of feature map dimensions between teacher and student — via 1×1 convolutional adapters. The adapter adds a few parameters to the student but remains small.

How to Distill a Model if Original Data Is Unavailable?

Sometimes the original dataset is unavailable (IP restrictions, privacy). Data-free distillation — generate synthetic data that maximizes teacher activations:

# DAFL (Data-Free Learning): generator creates "samples" for distillation
generator = Generator(latent_dim=256, img_channels=3)
optimizer_G = torch.optim.Adam(generator.parameters(), lr=1e-4)

for step in range(1000):
    z = torch.randn(batch_size, 256)
    fake_images = generator(z)

    # Losses: maximize teacher confidence + minimize BatchNorm statistics mismatch
    teacher_out = teacher(fake_images)
    activation_loss = -teacher_out.max(dim=1)[0].mean()  # teacher should be confident

    # BN statistics matching
    bn_loss = compute_bn_statistics_loss(teacher, fake_images)

    total_loss = activation_loss + 0.1 * bn_loss

    optimizer_G.zero_grad()
    total_loss.backward()
    optimizer_G.step()

The quality of data-free distillation is lower than the full-data version, but sometimes it is the only option.

Distillation for NLP Tasks on Mobile

For mobile apps with NLP (sentiment classification, intent detection, summarization): distill from GPT-4 / Claude API responses into a small BERT/DistilBERT.

# Collect soft labels from teacher (GPT-4 API)
# For each training example request class probabilities
# Store as training labels for student
# Student — DistilBERT fine-tuned on these soft labels

DistilBERT (66 MB, ONNX int8 — 18 MB) runs on-device in 30–80 ms on iOS/Android. GPT-4 in the cloud — hundreds of ms, costs per request.

What's Included in Our Work

• Resource budget analysis: RAM, CPU, GPU, power consumption
• Student architecture selection based on your constraints
• Teacher preparation (fine-tuning, quantization)
• Distillation with hyperparameter tuning (temperature, alpha, intermediate layers)
• Validation on test set and on real devices
• Conversion to Core ML / TFLite with int8 quantization
• Integration and support (updates for new OS versions)

Our team has 10+ years of experience in mobile development and ML. We guarantee the model will run stably on iOS 14+ and Android 10+. Get a consultation from an engineer on your model. Order distillation — we'll prepare a preliminary estimate in 1 day.

Timeline Estimates

Basic logit distillation for a classification task — 2–4 weeks (GPU time plus hyperparameter tuning). Full distillation with intermediate layers, non-standard architectures, data augmentation — 5–10 weeks. Custom estimate upon request.

Further reading: Knowledge Distillation on Wikipedia, PyTorch tutorial.