AI Food Photo Recognition & Calorie Counting in Mobile Apps

Food Recognition and Calorie Counting from Photos: AI in Mobile Apps

A user snaps a bowl of borscht—the app must output the macros. In practice, this is a chain: detection, classification, portion estimation, database lookup. Each link introduces error. Recently, a US startup came to us: their MVP recognized only 10 dishes with 70% accuracy. After implementing our pipeline, accuracy rose to 92%, and response time dropped from 2 seconds to 400 ms. We've accumulated experience in this domain across dozens of projects—from MVPs for startups to full fitness apps for iOS and Android, using Swift, Kotlin, Flutter, CoreML, and TFLite. Our track record: 10+ years in mobile AI, 40+ projects delivered.

When properly configured, recognition accuracy stays above 85% for the top 50 dishes. Delivery time for a basic version: 1 to 2 weeks. In this article, we break down the technical architecture, compare APIs, show code for iOS and Android, explain how to avoid common pitfalls, and integrate HealthKit with Health Connect.

How Is the Pipeline Built?

The right architecture isn't "one model to rule them all"—it's a pipeline of specialized steps.

Step 1: Detection and Classification

CoreML on iOS (model based on EfficientDet or YOLOv8), TFLite on Android. For an MVP, cloud APIs are viable: Clarifai Food Model, Google Cloud Vision with food tags, or the dedicated Logmeal API.

Step 2: Portion Estimation

This is harder. Without a reference object in the frame (coin, hand, standard plate), estimating grams is nearly impossible. Two practical solutions: ask the user to specify the container type (20 cm plate, 200 ml glass) or use ARKit/ARCore for depth estimation. Depth estimation via ARKit gives acceptable results for voluminous dishes—error margin 15–25%, which beats manual user input (they usually underestimate).

Step 3: Nutritional Data Lookup

USDA FoodData Central is a free API with 700,000+ products. Open Food Facts is an open-source database, good for packaged foods. For Russian-language markets, it's critical to have local dishes: borscht, pelmeni, Olivier—they aren't in USDA in the expected format. We use a combination of databases and supplement with custom data.

// iOS: Full recognition pipeline struct FoodRecognitionPipeline { func analyze(image: UIImage, portionContext: PortionContext?) async throws -> MealAnalysis { // 1. Dish recognition via Logmeal API let foodItems = try await logmealClient.recognizeFood(image: image) // 2. Portion estimation let portionEstimates: [PortionEstimate] if let context = portionContext { portionEstimates = estimatePortionFromContext(foodItems, context: context) } else { portionEstimates = try await estimatePortionWithAR(image: image) } // 3. Nutritional data let nutritionData = try await withThrowingTaskGroup(of: NutritionResult.self) { group in for (item, portion) in zip(foodItems, portionEstimates) { group.addTask { try await self.fetchNutrition(food: item, grams: portion.estimatedGrams) } } return try await group.reduce(into: []) { $0.append($1) } } return MealAnalysis( items: foodItems, portions: portionEstimates, nutrition: nutritionData.aggregate(), confidence: foodItems.map(\.confidence).min() ?? 0 ) } } 

Parallel nutritional requests via TaskGroup are crucial: with 3 dishes in the frame, sequential requests cause 3× latency.

How to Estimate Portion Without a Reference Object?

If there's no coin or standard plate in the frame, use ARKit to estimate scene depth. On iOS, an ARSession and depth image suffice. The error margin of 15–25% is acceptable for daily tracking. An alternative is to ask the user to select a container type from presets. With proper calibration, portion estimation accuracy stays above 70%.

Composite Dishes: Approaches to Recognition

A photo of borscht contains beets, cabbage, carrots, potatoes, meat, and sour cream in unknown proportions. Two solutions:

Recipe database. An LLM or custom model breaks the dish into ingredients based on a recipe. Works for standard dishes, poorly for home cooking with variations.

User correction. After automatic recognition, the user sees the presumed composition and can remove or add ingredients. Swipe-to-remove on an ingredient, slider for grams. This UX is fundamentally better than "98% accuracy" without editability.

// Android: Dish composition UI with editing @Composable fun MealCompositionEditor( items: List<FoodItem>, onItemRemoved: (FoodItem) -> Unit, onPortionChanged: (FoodItem, Float) -> Unit ) { LazyColumn { items(items, key = { it.id }) { item -> SwipeToDismiss( state = rememberDismissState { if (it == DismissValue.DismissedToStart) { onItemRemoved(item); true } else false }, background = { DeleteBackground() }, dismissContent = { FoodItemRow( item = item, onPortionChange = { grams -> onPortionChanged(item, grams) } ) } ) } } } 

Integration with HealthKit and Health Connect

A logged meal should enter the health ecosystem. We follow Apple's recommendations for HealthKit.

// iOS: Log to HealthKit func logMealToHealthKit(_ meal: MealAnalysis) async throws { let store = HKHealthStore() let caloriesType = HKQuantityType(.dietaryEnergyConsumed) let proteinType = HKQuantityType(.dietaryProtein) let carbsType = HKQuantityType(.dietaryCarbohydrates) let fatType = HKQuantityType(.dietaryFatTotal) let metadata: [String: Any] = [ HKMetadataKeyFoodType: meal.primaryItem?.name ?? "Mixed Meal" ] let samples = [ HKQuantitySample(type: caloriesType, quantity: .init(unit: .kilocalorie(), doubleValue: meal.nutrition.calories), start: .now, end: .now, metadata: metadata), HKQuantitySample(type: proteinType, quantity: .init(unit: .gram(), doubleValue: meal.nutrition.protein), start: .now, end: .now) // + carbs, fat ] try await store.save(samples) } 

Permissions are requested in advance via HKHealthStore.requestAuthorization. A common mistake is to request permissions on first app launch, before the user sees value. We recommend requesting at the moment of the first meal log—conversion increases by 30%.

Accuracy Barriers and How to Honestly Display Them

Even a good model fails on non-standard dishes, poor lighting, and unusual angles. Hiding uncertainty is a mistake. Show a confidence score next to the result.

struct NutritionDisplayView: View { let analysis: MealAnalysis var body: some View { VStack(alignment: .leading, spacing: 12) { if analysis.confidence < 0.6 { ConfidenceWarningBanner( message: "Low recognition confidence. Please verify the dish composition." ) } CalorieSummaryCard(nutrition: analysis.nutrition) MacroBreakdownChart(nutrition: analysis.nutrition) IngredientList(items: analysis.items, editable: true) } } } 

How to Integrate Recognition: 5 Steps

1. Choose a recognition API. Determine which service best fits your audience: Logmeal for high accuracy, Clarifai for quick start, Google Cloud Vision for versatility.
2. Adapt the model to the platform. Convert to CoreML (iOS) or TFLite (Android) for on-device operation. Provide a fallback to cloud API when confidence is low.
3. Implement portion estimation. Integrate ARKit/ARCore for automatic estimation or contextual container presets for manual input.
4. Connect a nutritional database. Link USDA FoodData Central for basic products, supplement with custom data for regional dishes.
5. Add HealthKit/Health Connect. Automate meal logging, requesting permissions at the moment of the first log.

What's Included in the Work

• Architecture design and API selection: analyze existing codebase, choose optimal recognition services and nutritional databases.
• Pipeline development: integrate detection, portion estimation, and data lookup—on iOS (Swift) and Android (Kotlin).
• UI/UX adaptation: customizable input screens, composition editing, confidence display.
• HealthKit/Health Connect integration: automatic meal logging into the health system.
• Documentation and support: code delivery, deployment instructions, test reports.

Time Estimates

Contact us for an evaluation of your project—we'll calculate exact timelines and cost. Request a consultation to discuss integration details with your app. Budget savings on development are achieved by using ready-made pipeline modules. Reducing AI function development costs is possible by choosing off-the-shelf APIs instead of training custom models.

Get a consultation—we'll help you select the optimal architecture for food recognition.