AI Image Editing (Inpainting) for Mobile App

Implementing AI Image Editing (Inpainting) in a Mobile App

Inpainting replaces a selected image region with new content generated from a prompt while preserving context of the rest. Remove a random passerby from a photo, change background behind portrait, add object to scene. Technically the task splits into three parts: create mask on device, send image + mask to API, display result.

Mask drawing: the most critical UX part

Mask is a grayscale image of the same size as original: white pixels — area for change, black — preserve.

On iOS — custom UIView with CALayer for drawing:

class MaskDrawingView: UIView {
    private var maskLayer = CAShapeLayer()
    private var path = UIBezierPath()
    private var brushSize: CGFloat = 30

    override func draw(_ rect: CGRect) {
        UIColor.black.setFill()
        UIRectFill(rect)
        UIColor.white.setFill()
        path.fill()
    }

    override func touchesMoved(_ touches: Set<UITouch>, with event: UIEvent?) {
        guard let touch = touches.first else { return }
        let point = touch.location(in: self)
        let circle = UIBezierPath(arcCenter: point, radius: brushSize / 2, startAngle: 0, endAngle: .pi * 2, clockwise: true)
        path.append(circle)
        setNeedsDisplay()
    }

    func getMaskImage() -> UIImage {
        UIGraphicsBeginImageContextWithOptions(bounds.size, false, 0)
        layer.render(in: UIGraphicsGetCurrentContext()!)
        let image = UIGraphicsGetImageFromCurrentImageContext()!
        UIGraphicsEndImageContext()
        return image
    }
}

On Android — Canvas + Paint in custom View:

class MaskDrawingView(context: Context) : View(context) {
    private val maskBitmap = Bitmap.createBitmap(width, height, Bitmap.Config.ARGB_8888)
    private val canvas = Canvas(maskBitmap)
    private val paint = Paint().apply {
        color = Color.WHITE
        style = Paint.Style.FILL
        strokeWidth = brushSize
        isAntiAlias = true
    }

    override fun onTouchEvent(event: MotionEvent): Boolean {
        when (event.action) {
            MotionEvent.ACTION_MOVE -> {
                canvas.drawCircle(event.x, event.y, brushSize / 2, paint)
                invalidate()
            }
        }
        return true
    }
}

Important: mask must be same resolution as original image. If user draws on 375×375 pt preview but original is 4032×3024 px — scale mask before sending.

API integration

DALL-E 2 Inpainting

func inpaint(image: UIImage, mask: UIImage, prompt: String) async throws -> UIImage {
    guard let imageData = image.pngData(), let maskData = mask.pngData() else {
        throw InpaintError.invalidImage
    }

    var request = URLRequest(url: URL(string: "https://api.openai.com/v1/images/edits")!)
    request.httpMethod = "POST"
    request.setValue("Bearer \(apiKey)", forHTTPHeaderField: "Authorization")

    let boundary = UUID().uuidString
    request.setValue("multipart/form-data; boundary=\(boundary)", forHTTPHeaderField: "Content-Type")

    var body = Data()
    // image (must be PNG, RGBA, max 4 MB)
    body.appendMultipart(boundary: boundary, name: "image", filename: "image.png", contentType: "image/png", data: imageData)
    // mask (PNG, RGBA, transparency = area for change)
    body.appendMultipart(boundary: boundary, name: "mask", filename: "mask.png", contentType: "image/png", data: maskData)
    // prompt
    body.appendMultipart(boundary: boundary, name: "prompt", data: prompt.data(using: .utf8)!)
    // size (must match input image size)
    body.appendMultipart(boundary: boundary, name: "size", data: "1024x1024".data(using: .utf8)!)
    body.append("--\(boundary)--\r\n".data(using: .utf8)!)
    request.httpBody = body

    let (data, _) = try await URLSession.shared.data(for: request)
    let response = try JSONDecoder().decode(ImageResponse.self, from: data)
    // Download result
    let (imageData2, _) = try await URLSession.shared.data(from: URL(string: response.data[0].url)!)
    return UIImage(data: imageData2)!
}

DALL-E 2 limitation: accepts only PNG with alpha channel (RGBA). Mask passed via transparency: transparent pixels = edit area. Not grayscale mask like in SD, but alpha channel. Max size — 4 MB. If image is JPEG — convert to PNG with alpha channel added.

Stable Diffusion Inpainting via Replicate

val body: [String: Any] = [
    "version": "...", // SD inpainting model
    "input": [
        "prompt": prompt,
        "image": "data:image/jpeg;base64,${imageBase64}",
        "mask": "data:image/png;base64,${maskBase64}",
        "num_inference_steps": 25,
        "guidance_scale": 7.5,
        "strength": 0.99 // 1.0 = full replacement, 0.5 = soft blend
    ]
]

SD inpainting via Replicate accepts base64 for image and mask. strength controls how much model deviates from original in mask area: 0.99 — nearly full replacement, 0.5 — blend with original.

Image transformation for API requirements

DALL-E 2 requires exactly 1024×1024 (or 256, 512). If user selected 4032×3024 photo — resize:

func resizeAndCrop(_ image: UIImage, to size: CGSize) -> UIImage {
    UIGraphicsBeginImageContextWithOptions(size, false, 1.0)
    let aspectFill = max(size.width / image.size.width, size.height / image.size.height)
    let newSize = CGSize(width: image.size.width * aspectFill, height: image.size.height * aspectFill)
    let origin = CGPoint(x: (size.width - newSize.width) / 2, y: (size.height - newSize.height) / 2)
    image.draw(in: CGRect(origin: origin, size: newSize))
    let result = UIGraphicsGetImageFromCurrentImageContext()!
    UIGraphicsEndImageContext()
    return result
}

After inpainting — if result needs return to original proportions, overlay result back on original image in mask coordinates.

Timeline

Mask drawing screen + inpainting via DALL-E 2 — 5–8 days. Full editor with mask undo/redo, brush scaling, result overlay preview, SD Inpainting — 3–4 weeks.