A typical scenario: a model trained on an 80 GB GPU won't run on a Raspberry Pi — latency in seconds, OOM at every inference. Optimization for edge devices is a set of techniques that reduce model size and latency while maintaining acceptable quality. This article covers the key methods we apply in projects and shows how to achieve 4–8x compression without significant accuracy loss.
A common mistake is believing that simply converting the model to TFLite suffices. Without device adaptation, accuracy drops by 10–15% and latency remains high. We use a combination of quantization, pruning, and distillation to achieve maximum acceleration on specific hardware.
Real case: face recognition on Jetson Nano. Original ResNet-50 (98 MB, Float32) ran at 800 ms per frame. After PTQ INT8, size dropped to 25 MB, latency to 150 ms. Accuracy loss was 0.3%. We then applied structured pruning (50% channels) — latency fell to 90 ms. Final speedup: 9x. This demonstrates that a pipeline of structured pruning followed by quantization yields a 9x better performance than the unoptimized model.
How to Optimize an ML Model for Edge?
Key techniques include quantization, pruning, knowledge distillation, and neural architecture search. Let's examine each.
Quantization
The most impactful method. Converting weights from Float32 to INT8 reduces size 4x and speeds up inference 2–4x on supporting hardware. INT4 yields 8x compression but higher accuracy loss. Post-Training Quantization (PTQ) requires a calibration dataset (100–1000 samples) and takes hours. Quantization-Aware Training (QAT) trains the model with quantization in mind, giving 1–3% better accuracy. We recommend QAT for critical tasks. See also Quantization in ML.
Pruning
Removing unimportant weights. Unstructured pruning can achieve 80%+ sparsity but is hard to accelerate on standard hardware without specialized libraries. Structured pruning (removing entire filters or heads) provides direct speedup on any device. In practice, we combine both: first unstructured pruning to 50%, then fine-tuning. Structured pruning is typically 2x better than unstructured in terms of latency reduction.
Distillation
A small student model is trained to reproduce the outputs of a large teacher model. Example: BERT → TinyBERT runs 7.5x faster while retaining 96% of GLUE score. Distillation is often combined with quantization for maximum effect. See Knowledge Distillation.
Neural Architecture Search
Finding the optimal architecture under target latency and memory constraints. MobileNetV2, found via NAS, became a standard for mobile devices. For edge projects, we use lightweight NAS based on regression models.
Operator Fusion
Merging sequential operations (Conv+BN+ReLU) into one. Implemented in TFLite converter, ONNX Runtime, TensorRT. Provides speedup without changing weights. This fusion yields a 10-20% better latency compared to unfused execution.
Which Techniques Give the Biggest Impact?
Compare the main approaches:
| Technique | Compression | Speedup | Accuracy Loss |
|---|---|---|---|
| PTQ INT8 | 4x | 2–4x | 0.5–2% |
| QAT INT8 | 4x | 2–4x | 0.1–0.5% |
| Unstructured pruning (50%) | 2x | 0–1x | 1–3% |
| Structured pruning (50%) | 2x | 1.5–2x | 1–3% |
| Distillation (Teacher→Student) | 2–4x | 2–7x | 1–5% |
Model compression examples:
| Model | Size (Float32) | Size (INT8) | Speedup |
|---|---|---|---|
| ResNet-50 | 98 MB | 25 MB | 2.5x |
| BERT-base | 440 MB | 110 MB | 3x |
| YOLOv8 | 200 MB | 50 MB | 2x |
When Should You Use Quantization-Aware Training?
QAT is justified when accuracy is critical and PTQ losses exceed 1%. For instance, in medical diagnostics or autonomous driving. We guarantee that with QAT, accuracy degrades no more than 0.5%. If a 2% loss is acceptable, PTQ is sufficient — faster and doesn't require access to training data. For a typical project, using QAT instead of PTQ costs an extra $500–$1000 but preserves an additional 1–2% accuracy — a worthwhile trade-off for high-stakes applications.
How to Combine Techniques for Maximum Speedup?
Often one method isn't enough. For a typical CV pipeline, we apply: structured pruning (remove 30% filters) → PTQ INT8 → operator fusion. For NLP: distill BERT into TinyBERT → QAT INT8. For detection: TensorRT with FP16 and INT8. Cost savings: instead of cloud inference, an edge device costing $100 once. We estimate payback in 3–6 months through reduced infrastructure costs. For example, switching from cloud to edge inference saves approximately $3,600 per year per device – a significant recurring reduction.
Process
The step-by-step process includes: 1) Model analysis: profile latency, memory, bottlenecks on the target device using layer-wise profiling. 2) Strategy selection: combine techniques for the specific task. For NLP, distillation + quantization. 3) Optimization: apply QAT or PTQ, pruning, fusion. 4) Validation: check accuracy on a representative dataset, compare with baseline. 5) Deployment: convert to TFLite, ONNX, or TensorRT format, integrate into the pipeline.
What's Included
- Optimized model in the format for your device (TFLite, ONNX, TensorRT)
- Profiling report with metric comparison
- Recommendations for further optimization
- Integration support
Timeline and Cost
Timelines: 2 to 4 weeks depending on model complexity and accuracy requirements. We'll assess your project for free — just contact us. Typical project cost ranges from $2,000 to $5,000, depending on the scope.
Our experience: 5+ years in AI/ML solution development, over 20 projects in edge optimization. We guarantee preservation of key quality metrics.
Order optimization — get a model that runs on Raspberry Pi, Jetson Nano, or any other edge device. Get a consultation for your project — we'll help select the optimal combination of techniques.