Shelf Monitoring: Computer Vision for Retail Shelf Audits

Shelf Monitoring: Computer Vision for Retail Shelf Audits

Shelf monitoring automates product placement checks in retail. We build computer vision systems that analyze each shelf every 15 minutes using existing cameras or autonomous robots. According to IHL Group, out-of-stock costs retailers $1 trillion annually globally — about 4% of revenue. Traditional inspections: employee walkthroughs take hours and provide a snapshot. Our automation reduces response time to minutes and increases accuracy to 95%.

Our experience — 12 years in CV for retail and 50+ implementations in Russia and CIS. We use YOLOv8 (accuracy mAP@50 — 0.92) and custom architectures for SKU recognition. In this article, we'll break down how we build such systems: from dataset collection to deployment on cameras or robots.

Why retailers lose billions on empty shelves?

Out-of-stock is not the only problem. Incorrect placement (planogram violation) reduces sales by 3–5%. Products not in their places — customers leave. Price tags mismatch — fines and dissatisfaction. Manual checks are expensive: one inspector on average visits 10 stores per week, spending 20 minutes per shelf. A chain of 200 stores means 400 person-hours per week just for shelf checks. A CV system does the same in 5 minutes per entire store, without interrupting operations. Each out-of-stock for a popular SKU costs on average $25 in lost profit.

How does product detection on the shelf work?

We use YOLOv8 — a state-of-the-art real-time detector. On custom datasets, it gives mAP@50 = 0.92 on typical sets, which is 15% higher than previous versions. The model is trained on images from various angles and lighting conditions. The code below shows a basic class for analyzing shelf images.

from ultralytics import YOLO
import numpy as np
import cv2

class ShelfMonitoringSystem:
    def __init__(self, product_model_path: str, planogram_path: str = None):
        self.detector = YOLO(product_model_path)
        self.planogram = self._load_planogram(planogram_path) if planogram_path else None

    def analyze_shelf_image(self, image: np.ndarray) -> dict:
        """Analysis of one shelf photo"""
        # Detection of all SKUs
        detections = self.detector(image, conf=0.4, iou=0.5)
        detected_products = []
        for box in detections[0].boxes:
            sku = self.detector.model.names[int(box.cls)]
            x1, y1, x2, y2 = map(int, box.xyxy[0])
            detected_products.append({
                'sku': sku,
                'bbox': [x1, y1, x2, y2],
                'confidence': float(box.conf),
                'facings': 1  # each bounding box = 1 facing
            })

        # Aggregation by SKU
        sku_counts = {}
        for product in detected_products:
            sku = product['sku']
            if sku not in sku_counts:
                sku_counts[sku] = {'count': 0, 'positions': []}
            sku_counts[sku]['count'] += 1
            sku_counts[sku]['positions'].append(product['bbox'])

        result = {
            'detected_skus': sku_counts,
            'total_facings': len(detected_products),
        }

        # Comparison with planogram
        if self.planogram:
            result['planogram_compliance'] = self._check_planogram(
                sku_counts, self.planogram
            )
            result['out_of_stock'] = self._find_out_of_stock(
                sku_counts, self.planogram
            )
        return result

    def _find_out_of_stock(self, current: dict, planogram: dict) -> list:
        """Finding missing products"""
        missing = []
        for sku, expected_facings in planogram.items():
            current_facings = current.get(sku, {}).get('count', 0)
            if current_facings == 0:
                missing.append({'sku': sku, 'status': 'out_of_stock', 'expected_facings': expected_facings})
            elif current_facings < expected_facings * 0.5:
                missing.append({'sku': sku, 'status': 'low_stock', 'current': current_facings, 'expected': expected_facings})
        return missing

Training the model on a catalog of 5000+ SKUs

For a large retailer, we use hierarchical classification: first determine the category, then the specific SKU. Each SKU requires at least 500 labeled images. Augmentations account for shelf specifics — rotations, scale, glare.

# Dataset structure: cropped product images by folder
# dataset/
# train/
# category_beverages/
# sku_cola_1l/
# sku_juice_orange/
# category_dairy/
# ...
from ultralytics import YOLO

model = YOLO('yolov8l.pt')
model.train(
    data='shelf_products.yaml',
    epochs=200,
    imgsz=640,
    batch=32,
    workers=8,
    optimizer='AdamW',
    lr0=1e-3,
    # Shelf-specific augmentations
    degrees=5.0,  # small rotation
    scale=0.3,  # scale change (different distances to shelf)
    fliplr=0.5,
    hsv_h=0.02,  # slight color change
    mosaic=1.0
)

Mobile app for inspectors

An employee takes a photo of the shelf via a mobile app, the system instantly shows:

• Green frame: SKU in stock, matches planogram
• Yellow frame: low stock
• Red frame: out of stock

class ShelfInspectionAPI:
    def __init__(self, monitor: ShelfMonitoringSystem):
        self.monitor = monitor

    def analyze_photo(self, image_bytes: bytes, store_id: str, shelf_id: str) -> dict:
        image = cv2.imdecode(
            np.frombuffer(image_bytes, np.uint8),
            cv2.IMREAD_COLOR
        )
        result = self.monitor.analyze_shelf_image(image)
        # Add visualization
        annotated = self._annotate_image(image, result)
        return {
            'store_id': store_id,
            'shelf_id': shelf_id,
            'analysis': result,
            'annotated_image_base64': encode_image_b64(annotated)
        }

Integration with robots for automated inspection

Autonomous robots (Simbe Tally, Brain Corp) patrol the store and photograph shelves. The CV system processes photos in real time, sends replenishment tasks to staff.

How to reduce losses from empty shelves?

Automation allows detecting out-of-stock 10 times faster than manual walkthroughs. The system compares the current state with the planogram and generates tasks for employees. A typical scenario: a robot patrols the store in one hour, the system identifies 15–20 violations, staff receives notifications on mobile devices. Response time — less than 5 minutes. This reduces revenue losses by 80%.

What is included in our work

1. Audit of current processes and cameras — assessment of image quality, lighting, viewing angles.
2. Collection and labeling of dataset — at least 5000 images per category, labeling quality control.
3. Model training with metrics mAP > 0.9 and recall > 0.85.
4. Integration with WMS/ERP (SAP, 1C, Oracle) via REST API.
5. Development of a mobile app for Android/iOS with online analysis.
6. Visualization of results in dashboards (Grafana, Power BI).
7. Documentation and training of the client's team.
8. 6 months of support, including retraining when new SKUs are added.

Timelines and Metrics

For small chains (up to 500 SKUs), implementation takes from 5 weeks; for large ones (5000+ SKUs), up to 16 weeks. We guarantee SKU detection accuracy of at least 88% and compliance check accuracy of 85%.

Our experience includes integrations with WMS SAP and 1C, as well as with Simbe Tally and Brain Corp robots. Certified engineers in PyTorch and TensorFlow. Over 5 years in the market, 12 years of expertise in computer vision.

Want to assess the feasibility of implementation? Contact us — we will conduct an audit of your cameras and provide a proof-of-concept in 2 weeks. Request a consultation to discuss your project.