AI Photo Colorization in Mobile Apps
Colorization is a task with one correct input and infinite "correct" outputs. Sky can be blue or gray. Coat can be black or blue. The model makes a statistically probable choice, not restores reality. Important to explain this via UX text, but doesn't change technical implementation.
Which Models to Use
Classic—DeOldify (fastai + U-Net with self-attention). Generalizes well, produces saturated colors. Downside: sometimes "stains" unwanted colors on faces or clothing with complex backgrounds.
DDColor (2023)—transformer-based architecture, performs much better on portraits and architecture. More accurate skin tone and sky transmission.
BigColor and ChromaGAN—for specific tasks (historical photos with heavy degradation).
Mobile runs one of them—converted to Core ML or TFLite. Original DeOldify weights ~250 MB, after INT8 quantization—60–70 MB. DDColor lighter—110 MB base version.
Converting DeOldify to Core ML
import coremltools as ct
import torch
from deoldify.visualize import get_image_colorizer
colorizer = get_image_colorizer(artistic=True) # or stable
model = colorizer.learn.model.eval()
# Export via torch.jit.trace
example = torch.zeros(1, 3, 256, 256)
traced = torch.jit.trace(model, example)
mlmodel = ct.convert(
traced,
inputs=[ct.TensorType(name="input", shape=(1, 3, 256, 256))],
compute_precision=ct.precision.FLOAT16,
minimum_deployment_target=ct.target.iOS16
)
mlmodel.save("DeOldify_artistic.mlpackage")
Important: DeOldify takes RGB input (even for grayscale—it converts to Lab internally and works with AB channels). Before passing grayscale image, expand to 3-channel RGB by replication: gray → [gray, gray, gray]. In CoreML this done in preprocessing, not in the model itself.
iOS: Running Inference
let config = MLModelConfiguration()
config.computeUnits = .cpuAndNeuralEngine // ANE for FLOAT16
let model = try DeOldify_artistic(configuration: config)
// Preparation: grayscale CVPixelBuffer → RGB
func prepareInput(from grayImage: UIImage) -> CVPixelBuffer? {
// Create RGB pixel buffer from grayscale, replicating channel
var pixelBuffer: CVPixelBuffer?
CVPixelBufferCreate(nil, width, height,
kCVPixelFormatType_32BGRA, nil, &pixelBuffer)
// ... copy gray channel to all three
return pixelBuffer
}
let input = DeOldify_artisticInput(input: pixelBuffer)
let output = try model.prediction(input: input)
let colorizedImage = UIImage(cvPixelBuffer: output.output)
On iPhone 13, 512×512 image processes in 0.8–1.2 seconds. For Full HD (1920×1080)—tiled inference with 512×512 patches and blending at seams. Tiled approach creates color inconsistency between patches: model might color one sky fragment blue, neighbor gray. Solution—global color histogram matching: normalize each tile histogram to global.
Android: TFLite with ONNX Runtime as Alternative
// ONNX Runtime gives flexibility: one model for iOS and Android
val env = OrtEnvironment.getEnvironment()
val session = env.createSession(
"deoldify_optimized.onnx",
OrtSession.SessionOptions().apply {
addNnapi() // or addCuda() if GPU available
}
)
val inputTensor = OnnxTensor.createTensor(env, inputArray, longArrayOf(1, 3, 512, 512))
val results = session.run(mapOf("input" to inputTensor))
val outputArray = (results[0].value as Array<*>)
ONNX Runtime Mobile—good choice when one model needed on both platforms: no need to convert twice. NNAPI delegate works on Android 8.1+.
UX: What Matters
Colorization slow operation (1–5 seconds). Show animated progress. Good pattern—"before/after" slider: user drags divider, sees original and result simultaneously. Both UX and clear model demo.
Saving result—in .heic or .jpg maximum quality. Colorization result poorly withstands re-JPEG compression: artifacts at color transitions become visible.
Process
Choose and convert model, optimize for target devices, implement tiled inference with color consistency, UI with before/after comparison, test on historical photos of different quality.
Timeline Estimates
One platform with basic colorization takes 2–4 weeks. Both platforms with tiling and color normalization requires 5–8 weeks.