Offline AI on Android: TensorFlow Lite with GPU/NNAPI Acceleration

Offline AI on Android: TensorFlow Lite with GPU/NNAPI Acceleration

We often encounter this situation: you have a trained object detection model in PyTorch and want to run it on a smartphone without internet. You convert it to TensorFlow Lite, add it to assets—and on a test Pixel 6 it flies. But on a Samsung Galaxy A21s (Exynos) the app crashes with OutOfMemoryError, and on a Xiaomi Redmi Note 8 (Qualcomm) it works but lags. The cause is the choice of acceleration delegate and memory management. TensorFlow Lite is the de facto standard for on-device ML, but its integration requires deep understanding of hardware specifics. Savings on cloud computing can reach 90%—thousands of dollars per month for a production service—but only with proper implementation. Our team has delivered solutions for 40+ projects, with average savings of $1500–$3000 per month per client. For one logistics client, we deployed a package detection model on Android scanners, cutting cloud inference costs from $5,000/month to near zero. The solution paid for itself in under 2 months.

TensorFlow Lite official documentation

How to Convert a Model with Minimal Loss?

The first step is to export the model from PyTorch/ONNX to TensorFlow. Then use TFLiteConverter with optimizations. For INT8, calibrate on a representative dataset. Example:

converter = tf.lite.TFLiteConverter.from_saved_model("model_tf")
converter.optimizations = [tf.lite.Optimize.DEFAULT]
# динамическая квантизация FP16
converter.target_spec.supported_types = [tf.float16]
# для GPU delegate
converter.representative_dataset = representative_dataset
# для INT8 (калибровка)
tflite_model = converter.convert()
with open("model_fp16.tflite", "wb") as f:
    f.write(tflite_model)

Why Delegate Choice Is Critical for Performance?

We use a hybrid configuration: GPU with fallback to XNNPACK and NNAPI as a last resort. Here’s how it looks in code:

import org.tensorflow.lite.gpu.GpuDelegate
import org.tensorflow.lite.gpu.CompatibilityList

val compatList = CompatibilityList()
val options = Interpreter.Options().apply {
    if (compatList.isDelegateSupportedOnThisDevice) {
        addDelegate(GpuDelegate(compatList.bestOptionsForThisDevice))
    } else {
        // Fallback: сначала NNAPI, если не сработает — XNNPACK
        setUseNNAPI(true)
        setUseXNNPACK(true)
    }
    setNumThreads(Runtime.getRuntime().availableProcessors())
}

var interpreter: Interpreter? = null
try {
    interpreter = Interpreter(FileUtil.loadMappedFile(context, "model_fp16.tflite"), options)
    // Тестовый прогон (нужен для выявления ошибок NNAPI)
    interpreter.run(testInput, testOutput)
} catch (e: Exception) {
    Log.w("ML", "NNAPI failed, fallback to CPU: ${e.message}")
    options.setUseNNAPI(false)
    interpreter = Interpreter(modelBuffer, options)
}

NNAPI is unstable in practice: on some devices it gives 5× speedup, on others it crashes. Always wrap the launch in try/catch. Without this, stability is not guaranteed.

Managing Buffers: ByteBuffer vs TensorBuffer

Direct ByteBuffer management is faster but verbose. TensorBuffer from org.tensorflow.lite.support is more convenient and less error-prone:

import org.tensorflow.lite.support.image.ImageProcessor
import org.tensorflow.lite.support.image.TensorImage
import org.tensorflow.lite.support.common.ops.NormalizeOp
import org.tensorflow.lite.support.image.ops.ResizeOp

val imageProcessor = ImageProcessor.Builder()
    .add(ResizeOp(224, 224, ResizeOp.ResizeMethod.BILINEAR))
    .add(NormalizeOp(127.5f, 127.5f))
    .build()

val tensorImage = TensorImage(DataType.FLOAT32)
tensorImage.load(bitmap)
val processedImage = imageProcessor.process(tensorImage)

val outputBuffer = TensorBuffer.createFixedSize(intArrayOf(1, 1000), DataType.FLOAT32)
interpreter.run(processedImage.buffer, outputBuffer.buffer)
val probabilities = outputBuffer.floatArray
val topIndex = probabilities.indices.maxByOrNull { probabilities[it] } ?: -1

CameraX Integration

val imageAnalyzer = ImageAnalysis.Builder()
    .setTargetResolution(Size(640, 480))
    .setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)
    .build()
    .also { it.setAnalyzer(cameraExecutor) { imageProxy ->
        try {
            val bitmap = imageProxy.toBitmap()
            runInference(bitmap)
        } finally {
            imageProxy.close() // КРИТИЧНО: иначе CameraX зависнет
        }
    } }

imageProxy.close() in a finally block is not optional. If you don't close the ImageProxy, CameraX stops delivering new frames after a few seconds.

Why Numerical Accuracy After Quantization Matters?

After conversion, always check accuracy on a test set. FP16 typically loses <1%, INT8 loses 1–3%. If losses are larger, the calibration dataset may be too small or the model sensitive to specific layers. Normal maximum deviation is 0.01 for FP16 and 0.05 for INT8. If deviation is higher, return to conversion: change optimizations or replace sensitive layers.

Performance Table on Different Chips (Example)

What the Work Includes

• Model conversion from your framework (PyTorch, ONNX, SavedModel) with optimization selection.
• Delegate selection and setup with fallback logic.
• Integration with CameraX or other data source.
• Numerical accuracy testing and profiling with Android Profiler + TFLite Benchmark Tool.
• Build testing on 10+ physical devices (different chips, Android versions).
• Documentation (API description, architecture), access to the code repository, developer training, and 2 weeks of support.

Typical Mistakes in TFLite Integration

• Forgetting to close ImageProxy—frame stream stops.
• Not checking delegate support on the device—crashes on old chips.
• Using INT8 without calibration—accuracy drop >10%.
• Loading the model into RAM without MappedByteBuffer—OOM on devices with 2 GB RAM.

Time Estimates

Basic TFLite model integration on Android takes 1–2 weeks. With multi-delegate logic, CameraX pipeline, testing on a device park—3–5 weeks. The cost is calculated individually based on complexity. Get a consultation for your task—we'll provide timelines and cost.

We are a team with in-house ML development experience, certified Google engineers. With 40+ projects behind us, including offline AI applications for automotive and medical industries. Contact us for an accurate estimate of your task—we'll send a plan and timeline.